Ten categories of statistical errors: a guide for research in endocrinology and metabolism

A simple framework is introduced that defines ten categories of statistical errors on the basis of type of error, bias or imprecision, and source: sampling, measurement, estimation, hypothesis testing, and reporting. Each of these ten categories is illustrated with examples pertinent to research and publication in the disciplines of endocrinology and metabolism. Some suggested remedies are discussed, where appropriate. A review of recent issues of American Journal of Physiology: Endocrinology and Metabolism and of Endocrinology finds that very small sample sizes may be the most prevalent cause of statistical error in this literature.     

Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results

Background

The widespread reluctance to share published research data is often hypothesized to be due to the authors'' fear that reanalysis may expose errors in their work or may produce conclusions that contradict their own. However, these hypotheses have not previously been studied systematically.

Methods and Findings

We related the reluctance to share research data for reanalysis to 1148 statistically significant results reported in 49 papers published in two major psychology journals. We found the reluctance to share data to be associated with weaker evidence (against the null hypothesis of no effect) and a higher prevalence of apparent errors in the reporting of statistical results. The unwillingness to share data was particularly clear when reporting errors had a bearing on statistical significance.

Conclusions

Our findings on the basis of psychological papers suggest that statistical results are particularly hard to verify when reanalysis is more likely to lead to contrasting conclusions. This highlights the importance of establishing mandatory data archiving policies.     

Ca mishandling and cardiac dysfunction in obesity and insulin resistance: Role of oxidative stress

Obesity and insulin resistance (IR) are strongly connected to the development of subclinical cardiac dysfunction and eventually can lead to heart failure, which is the main cause of morbidity and death in patients having these metabolic diseases. It has been considered that excessive fat tissue may play a critical role in producing systemic IR and enhancing reactive oxygen species (ROS) generation. This oxidative stress (OS) may elicit or exacerbate IR. On the other hand, evidence suggests that some of the cellular mechanisms involved in the pathophysiology of obesity and IR-related cardiomyopathy are excessive myocardial ROS production and abnormal Ca²⁺ homeostasis. In addition, emerging evidence suggests that augmented ROS production may contribute to Ca²⁺ mishandling by affecting the redox state of key proteins implicated in this process. In this review, we focus on the role of Ca²⁺ mishandling in the development of cardiac dysfunction in obesity and IR and address the evidence suggesting that OS might also contribute to cardiac dysfunction by affecting Ca²⁺ handling.     

Outlier Removal and the Relation with Reporting Errors and Quality of Psychological Research

Background

The removal of outliers to acquire a significant result is a questionable research practice that appears to be commonly used in psychology. In this study, we investigated whether the removal of outliers in psychology papers is related to weaker evidence (against the null hypothesis of no effect), a higher prevalence of reporting errors, and smaller sample sizes in these papers compared to papers in the same journals that did not report the exclusion of outliers from the analyses.

Methods and Findings

We retrieved a total of 2667 statistical results of null hypothesis significance tests from 153 articles in main psychology journals, and compared results from articles in which outliers were removed (N = 92) with results from articles that reported no exclusion of outliers (N = 61). We preregistered our hypotheses and methods and analyzed the data at the level of articles. Results show no significant difference between the two types of articles in median p value, sample sizes, or prevalence of all reporting errors, large reporting errors, and reporting errors that concerned the statistical significance. However, we did find a discrepancy between the reported degrees of freedom of t tests and the reported sample size in 41% of articles that did not report removal of any data values. This suggests common failure to report data exclusions (or missingness) in psychological articles.

Conclusions

We failed to find that the removal of outliers from the analysis in psychological articles was related to weaker evidence (against the null hypothesis of no effect), sample size, or the prevalence of errors. However, our control sample might be contaminated due to nondisclosure of excluded values in articles that did not report exclusion of outliers. Results therefore highlight the importance of more transparent reporting of statistical analyses.     

Statistical Reporting Errors and Collaboration on Statistical Analyses in Psychological Science

Statistical analysis is error prone. A best practice for researchers using statistics would therefore be to share data among co-authors, allowing double-checking of executed tasks just as co-pilots do in aviation. To document the extent to which this ‘co-piloting’ currently occurs in psychology, we surveyed the authors of 697 articles published in six top psychology journals and asked them whether they had collaborated on four aspects of analyzing data and reporting results, and whether the described data had been shared between the authors. We acquired responses for 49.6% of the articles and found that co-piloting on statistical analysis and reporting results is quite uncommon among psychologists, while data sharing among co-authors seems reasonably but not completely standard. We then used an automated procedure to study the prevalence of statistical reporting errors in the articles in our sample and examined the relationship between reporting errors and co-piloting. Overall, 63% of the articles contained at least one p-value that was inconsistent with the reported test statistic and the accompanying degrees of freedom, and 20% of the articles contained at least one p-value that was inconsistent to such a degree that it may have affected decisions about statistical significance. Overall, the probability that a given p-value was inconsistent was over 10%. Co-piloting was not found to be associated with reporting errors.     

Seeing the forest beyond the trees

In a recent paper (Mitchard et al. 2014, Global Ecology and Biogeography, 23 , 935–946) a new map of forest biomass based on a geostatistical model of field data for the Amazon (and surrounding forests) was presented and contrasted with two earlier maps based on remote‐sensing data Saatchi et al. (2011; RS1) and Baccini et al. (2012; RS2). Mitchard et al. concluded that both the earlier remote‐sensing based maps were incorrect because they did not conform to Mitchard et al. interpretation of the field‐based results. In making their case, however, they misrepresented the fundamental nature of primary field and remote‐sensing data and committed critical errors in their assumptions about the accuracy of research plots, the interpolation methodology and the statistical analysis. By ignoring the large uncertainty associated with ground estimates of biomass and the significant under‐sampling and spatial bias of research plots, Mitchard et al. reported erroneous trends and artificial patterns of biomass over Amazonia. Because of these misrepresentations and methodological flaws, we find their critique of the satellite‐derived maps to be invalid.     

The geographic distribution of obesity in the US and the potential regional differences in misreporting of obesity

Objective: State‐level estimates of obesity based on self‐reported height and weight suggest a geographic pattern of greater obesity in the Southeastern US; however, the reliability of the ranking among these estimates assumes errors in self‐reporting of height and weight are unrelated to geographic region. Design and Methods: Regional and state‐level prevalence of obesity (body mass index ≥ 30 kg m^?2) for non‐Hispanic black and white participants aged 45 and over were estimated from multiple sources: ( 1 ) self‐reported from the behavioral risk factor surveillance system (BRFSS 2003‐2006) (n = 677,425), ( 2 ) self‐reported and direct measures from the National Health and Nutrition Examination Study (NHANES 2003‐2008) (n = 6,615 and 6,138, respectively), and ( 3 ) direct measures from the REasons for Geographic and Racial Differences in Stroke (REGARDS 2003‐2007) study (n = 30,239). Results: Data from BRFSS suggest that the highest prevalence of obesity is in the East South Central Census division; however, direct measures suggest higher prevalence in the West North Central and East North Central Census divisions. The regions relative ranking of obesity prevalence differs substantially between self‐reported and directly measured height and weight. Conclusions: Geographic patterns in the prevalence of obesity based on self‐reported height and weight may be misleading, and have implications for current policy proposals.     

Variable reporting and quantitative reviews: a comparison of three meta-analytical techniques

Variable reporting of results can influence quantitative reviews by limiting the number of studies for analysis, and thereby influencing both the type of analysis and the scope of the review. We performed a Monte Carlo simulation to determine statistical errors for three meta‐analytical approaches and related how such errors were affected by numbers of constituent studies. Hedges’d and effect sizes based on item response theory (IRT) had similarly improved error rates with increasing numbers of studies when there was no true effect, but IRT was conservative when there was a true effect. Log response ratio had low precision for detecting null effects as a result of overestimation of effect sizes, but high ability to detect true effects, largely irrespective of number of studies. Traditional meta‐analysis based on Hedges’d are preferred; however, quantitative reviews should use various methods in concert to improve representation and inferences from summaries of published data.     

Quantitative approaches in climate change ecology

Contemporary impacts of anthropogenic climate change on ecosystems are increasingly being recognized. Documenting the extent of these impacts requires quantitative tools for analyses of ecological observations to distinguish climate impacts in noisy data and to understand interactions between climate variability and other drivers of change. To assist the development of reliable statistical approaches, we review the marine climate change literature and provide suggestions for quantitative approaches in climate change ecology. We compiled 267 peer‐reviewed articles that examined relationships between climate change and marine ecological variables. Of the articles with time series data (n = 186), 75% used statistics to test for a dependency of ecological variables on climate variables. We identified several common weaknesses in statistical approaches, including marginalizing other important non‐climate drivers of change, ignoring temporal and spatial autocorrelation, averaging across spatial patterns and not reporting key metrics. We provide a list of issues that need to be addressed to make inferences more defensible, including the consideration of (i) data limitations and the comparability of data sets; (ii) alternative mechanisms for change; (iii) appropriate response variables; (iv) a suitable model for the process under study; (v) temporal autocorrelation; (vi) spatial autocorrelation and patterns; and (vii) the reporting of rates of change. While the focus of our review was marine studies, these suggestions are equally applicable to terrestrial studies. Consideration of these suggestions will help advance global knowledge of climate impacts and understanding of the processes driving ecological change.     

Performance of different methods for testing polymyxin B: comparison of broth microdilution,agar dilution and MIC test strip in mcr-1 positive and negative Escherichia coli

Antimicrobial susceptibility testing with the last-resort antibiotics polymyxins (polymyxin B and colistin) is associated with several methodological issues. Currently, broth microdilution (BMD) is recommended for colistin and polymyxin B. BMD is laborious and the utility of alternative methods needs to be evaluated for polymyxin B susceptibility testing. In this study, using BMD as a reference method, the performance of agar dilution (AD) and MIC test strips (MTS) were evaluated in polymyxin B susceptibility testing. BMD, AD and MTS were used to determine MICs of 193 clinical isolates of Escherichia coli. Seventy-nine were positive for the polymyxin resistance gene mcr-1. Method performances were evaluated based on pair-wise agreements with the reference method (BMD) and statistical testing. AD and MTS showed an unacceptable number of very major errors (VMEs) compared with BMD, 9·3 and 10·7%, respectively. The essential agreement (EA) was low for AD (49·7%), but high for MTS (97·8%). However, statistical testing showed that MTS tended to yield a one-step lower MIC (P < 0·01) compared with BMD. The discordances observed with MTS and AD in comparison with BMD for polymyxin B susceptibility testing for E. coli suggest their inapplicability in routine testing. A large number of isolates clustered around the susceptibility breakpoint (2–4 mg l⁻¹) and several mcr-1 positive isolates (17%) were determined as susceptible with BMD. A screening breakpoint for mcr-1 of 2 mg l⁻¹ should also be considered.     

Ethnic Differences in Self‐reported and Measured Obesit

As use of self‐reported data to classify obesity continues, ethnic differences in reporting errors remain unclear. The objective of this study is to elucidate misreporting disparities between African Americans (AAs) and European Americans (EAs). The Pennington Center Longitudinal Study (PCLS) is an ongoing investigation of environmental, behavioral, and biological factors associated with obesity, diabetes, and other common diseases. Self‐reported and measured height and weight were collected during initial screening for eligibility in various studies by telephone and clinic visits. All ethnicity‐sex groups (15,656 adults aged 18–65 years, 53% obese, 34% AA, 37% men) misreported heights and weights increasingly as measured values increased (P < 0.0001). More AA vs. EA women (P < 0.001) misreported height and weight, but more EA vs. AA men misreported their weight (P < 0.02). Obesity was underestimated more in AA vs. EA women (self‐reported ? measured prevalence = ?4.0% (AA) vs. ?2.6% (EA), P < 0.0001), but less in AA vs. EA men (?3.2% (AA) vs. ?4.2% (EA), P < 0.0001)). With measured obesity prevalence equalized at 53% in all groups, the self‐reported obesity prevalence in women was 50.4% (AA) vs. 49.6% (EA), and in men 49.8% (AA) vs. 47.3 (EA). Underestimation in women was ?2.6% (AA) vs. ?3.4% (EA); in men it was ?3.2% (AA) vs. ?5.7% (EA), P < 0.003. Self‐reported height and weight portend underestimation of obesity prevalence and the effect varies by ethnicity and gender. However, comparisons depend on the true prevalence within ethnicity‐gender groups. After controlling for obesity prevalence, disparity in underestimation was greater in EA than in AA men (P < 0.003) but not women.     

Reproductive biology of the European Cuckoo Cuculus canorus: early insights, persistent errors and the acquisition of knowledge

The brood parasitic habits of the European Cuckoo Cuculus canorus have excited wonder, disbelief and speculation since the fourth century BC. Accurate knowledge of cuckoo biology, however, accumulated only slowly and mostly since 1700. The aim of this study is to review six main topics: (1) the placement of cuckoo eggs in host nests; (2) cuckoo ‘clutch’ size; (3) cuckoo egg characteristics, mimicry and rejection; (4) choice of hosts; (5) eviction of eggs and chicks; and (6) the reasons why cuckoos are brood parasites and are incapable of rearing their own young. Early errors in reporting cuckoo biology were often a consequence of poor or incomplete observations leading to erroneous interpretations. Many of the early observers were egg collectors who focussed almost exclusively on the egg-laying period, thus ignoring cuckoo chick biology. Major landmarks in cuckoo studies included the facts that: (1) cuckoo eggs often resembled those of their hosts (1760s) and that this mimicry was adaptive (1850s); (2) hosts sometimes evicted cuckoo eggs (1770s); (3) female cuckoos laid individually distinctive eggs and that specific cuckoo gentes may exist (1850s); and (4) although well recognised that cuckoo chicks were reared alone, prior to Jenner’s work in the 1780s female cuckoo parents were thought to either eat or evict the host eggs or young. Jenner’s results was more readily accepted in Britain than in Germany. Between 1700 and 1859, cuckoo brood parasitism was difficult to reconcile with the prevalent conceptual framework of physico-theology (later known as the argument from design). Thereafter, Darwin’s idea of natural selection provided a superior conceptual framework, which in conjunction with experimental testing of specific hypotheses has continued to advance our understanding of brood parasitism. Our knowledge of cuckoo biology is far from complete, however, and we predict that continuing research often incorporating new technologies will refine and extend our understanding of the cuckoo’s extraordinary biology.     

Understanding ‘it depends’ in ecology: a guide to hypothesising,visualising and interpreting statistical interactions

Ecologists routinely use statistical models to detect and explain interactions among ecological drivers, with a goal to evaluate whether an effect of interest changes in sign or magnitude in different contexts. Two fundamental properties of interactions are often overlooked during the process of hypothesising, visualising and interpreting interactions between drivers: the measurement scale – whether a response is analysed on an additive or multiplicative scale, such as a ratio or logarithmic scale; and the symmetry – whether dependencies are considered in both directions. Overlooking these properties can lead to one or more of three inferential errors: misinterpretation of (i) the detection and magnitude (Type-D error), and (ii) the sign of effect modification (Type-S error); and (iii) misidentification of the underlying processes (Type-A error). We illustrate each of these errors with a broad range of ecological questions applied to empirical and simulated data sets. We demonstrate how meta-analysis, a widely used approach that seeks explicitly to characterise context dependence, is especially prone to all three errors. Based on these insights, we propose guidelines to improve hypothesis generation, testing, visualisation and interpretation of interactions in ecology.     

Adult obesity does not predict 6-year weight gain in men: the Aerobics Center Longitudinal Study

Objective: To determine the longitudinal relation between history of adult obesity and the 6‐year trajectory of weight change in men. Research Methods and Procedures: Subjects were healthy, affluent men (n = 761) between the ages of 20 and 78 years who completed at least four comprehensive medical exams at the Cooper Clinic between 1987 and 2003. Maximum adult weight was reported, and current height was measured at baseline. Body weight and cardiorespiratory fitness were measured at all examinations. Adult obesity status was determined from self‐reported maximum weight and measured height at baseline as BMI ≥ 30 kg/m². Weight at all examinations was regressed on a history of adult obesity using linear mixed effects modeling. Results: At baseline, men reporting a history of adult obesity were significantly heavier than men reporting no such history (BMI 29.8 vs. 25.0 kg/m²; p < 0.05). However, the rate of weight gain among men with a history of obesity was slower than among men without a history of adult obesity (0.04 vs. 0.18 kg/yr; p = 0.09), although this difference was only marginally significant. Fitness modulated the relationship between history of obesity and weight change over time, and both higher levels of fitness and greater frequency of dieting were associated with attenuated weight gain. In contrast, chronic disease and depression were associated with accelerated weight gain. Discussion: Although a history of obesity was associated with higher weight, it did not seem to result in accelerated weight gain over time. Additionally, dieting and fitness were important for minimizing weight gain.     

Paired versus two-group experimental design for rheological studies of vocal fold tissues

Vibratory function of the vocal folds is largely determined by the rheological properties or viscoelastic shear properties of the vocal fold lamina propria. To date, investigation of the sample size estimation and statistical experimental design for vocal fold rheological studies is nonexistent. The current work provides the closed-form sample size formulas for two major study designs (i.e. paired and two-group designs) in vocal fold research. Our results demonstrated that the paired design could greatly increase the statistical power compared to the two-group design. By comparing the variance of estimated treatment effect, this study also confirms that ignoring within-subject and within-vocal fold correlations during rheological data analysis will likely increase type I errors. Finally, viscoelastic shear properties of intact and scarred rabbit vocal fold lamina propria were measured and used to illustrate theoretical findings in a realistic scenario and project sample size requirement for future studies.     

Two Years Later: Journals Are Not Yet Enforcing the ARRIVE Guidelines on Reporting Standards for Pre-Clinical Animal Studies

There is growing concern that poor experimental design and lack of transparent reporting contribute to the frequent failure of pre-clinical animal studies to translate into treatments for human disease. In 2010, the Animal Research: Reporting of In Vivo Experiments (ARRIVE) guidelines were introduced to help improve reporting standards. They were published in PLOS Biology and endorsed by funding agencies and publishers and their journals, including PLOS, Nature research journals, and other top-tier journals. Yet our analysis of papers published in PLOS and Nature journals indicates that there has been very little improvement in reporting standards since then. This suggests that authors, referees, and editors generally are ignoring guidelines, and the editorial endorsement is yet to be effectively implemented.     

In focus: Film,focus groups and working children in Bangladesh

The use of visuals in anthropological research is an established though much debated practice, both as a research tool and as a means of reporting. Pile sorts, mapping, thematic drawing, photographs, visual scales and pictorial triad testing are all visual methods that have been used in participatory and conventional ethnographic research to encourage discussion among study participants and to clarify detail. Our experience in the use of visual tools in a study conducted in 1997–98, among former child garment workers in Bangladesh, reinforces the value of the use of visuals in research. A documentary film was used in focus groups with children, most aged 10–13. The results suggest that film is a powerfully evocative tool and, combined with focus groups, is an excellent qualitative research technique. The research experience in Bangladesh also suggests that children are able to participate meaningfully in the research not in spite of but because of the use of documentary film.     

Wise use of statistical tools in ecological field studies

The currently dominating hypothetico-deductive research paradigm for ecology has statistical hypothesis testing as a basic element. Classic statistical hypothesis testing does, however, present the ecologist with two fundamental dilemmas when field data are to be analyzed: (1) that the statistically motivated demand for a random and representative sample and the ecologically motivated demand for representation of variation in the study area cannot be fully met at the same time; and (2) that the statistically motivated demand for independence of errors calls for sampling distances that exceed the scales of relevant pattern-generating processes, so that samples with statistically desirable properties will be ecologically irrelevant. Reasons for these dilemmas are explained by consideration of the classic statistical Neyman-Pearson test procedure, properties of ecological variables, properties of sampling designs, interactions between properties of the ecological variables and properties of sampling designs, and specific assumptions of the statistical methods. Analytic solutions to problems underlying the dilemmas are briefly reviewed. I conclude that several important research objectives cannot be approached without subjective elements in sampling designs. I argue that a research strategy entirely based on rigorous statistical testing of hypotheses is insufficient for field ecological data and that inductive and deductive approaches are complementary in the process of building ecological knowledge. I recommend that great care is taken when statistical tests are applied to ecological field data. Use of less formal modelling approaches is recommended for cases when formal testing is not strictly needed. Sets of recommendations, “Guidelines for wise use of statistical tools”, are proposed both for testing and for modelling. Important elements of wise-use guidelines are parallel use of methods that preferably belong to different methodologies, selection of methods with few and less rigorous assumptions, conservative interpretation of results, and abandonment of definitive decisions based a predefined significance level.     

Correlates of BMI misreporting among apparently healthy individuals: the ATTICA study

Objective: The aim of this study was to investigate correlates of misreporting in BMI, based on self‐reported weight and height, in a randomly selected population sample of Greek adults and to evaluate the effect of obesity status misclassification on the associations between obesity and disease. Research Methods and Procedures: During 2001 to 2002, we randomly enrolled 1514 men (18 to 87 years old) and 1528 women (18 to 89 years old) from the Attica area, Greece; the sampling was stratified by the age‐sex distribution of the region. Various sociodemographic, clinical, and psychological characteristics were self‐reported, and weight and height were measured and recorded in all participants. Results: The proportions of true positives and true negatives for correct obesity status identification were 62% and 97%, respectively. Women were 9 times more likely to be under‐reporters than men, whereas men were 7.5 times more likely to be over‐reporters. A 10‐year increase in age was associated with a 48% higher likelihood of being an under‐reporter and 26% lower likelihood of being an over‐reporter, irrespective of sex and other characteristics of the participants. Clinical status, such as the presence of hypertension and diabetes, was associated with under‐reporting of body weight. Furthermore, the use of self‐reported data may substantially exaggerate associations between obesity and obesity‐related diseases, such as diabetes, hypercholesterolemia, and hypertension. Discussion: The study indicates that, apart from age and sex, disease status may be another factor that influences misreporting of obesity status, with diabetic and hypertensive people to be more likely to under‐report their overweight. Use of self‐reported data may bias obesity—disease associations.     

Short sleep duration is associated with reduced leptin levels and increased adiposity: Results from the Quebec family study

Objective: To explore cross‐sectional associations between short sleep duration and variations in body fat indices and leptin levels during adulthood in a sample of men and women involved in the Québec Family Study. Research Methods and Procedures: Anthropometric measurements, plasma lipid‐lipoprotein profile, plasma leptin concentrations, and total sleep duration were determined in a sample of 323 men and 417 women ages 21 to 64 years. Results: When compared with adults reporting 7 to 8 hours of sleep per day, the adjusted odds ratio for overweight/obesity was 1.38 (95% confidence interval, 0.89 to 2.10) for those with 9 to 10 hours of sleep and 1.69 (95% confidence interval, 1.15 to 2.39) for those with 5 to 6 hours of sleep, after adjustment for age, sex, and physical activity level. In each sex, we observed lower adiposity indices in the 7‐ to 8‐hour sleeping group than in the 5‐ to 6‐hour sleeping group. However, all of these significant differences disappeared after statistical adjustment for plasma leptin levels. Finally, the well‐documented regression of plasma leptin levels over body fat mass was used to predict leptin levels of short‐duration sleepers (5 and 6 hours of sleep), which were then compared with their measured values. As expected, the measured leptin values were significantly lower than predicted values. Discussion: There may be optimal sleeping hours at which body weight regulation is facilitated. Indeed, short sleep duration predicts an increased risk of being overweight/obese in adults and is related to a reduced circulating leptin level relative to what is predicted by fat mass. Because sleep duration is a potentially modifiable risk factor, these findings might have important clinical implications for the prevention and treatment of obesity.