The continuing misuse of null hypothesis significance testing in biological anthropology

There is over 60 years of discussion in the statistical literature concerning the misuse and limitations of null hypothesis significance tests (NHST). Based on the prevalence of NHST in biological anthropology research, it appears that the discipline generally is unaware of these concerns. The p values used in NHST usually are interpreted incorrectly. A p value indicates the probability of the data given the null hypothesis. It should not be interpreted as the probability that the null hypothesis is true or as evidence for or against any specific alternative to the null hypothesis. P values are a function of both the sample size and the effect size, and therefore do not indicate whether the effect observed in the study is important, large, or small. P values have poor replicability in repeated experiments. The distribution of p values is continuous and varies from 0 to 1.0. The use of a cut‐off, generally p ≤ 0.05, to separate significant from nonsignificant results, is an arbitrary dichotomization of continuous variation. In 2016, the American Statistical Association issued a statement of principles regarding the misinterpretation of NHST, the first time it has done so regarding a specific statistical procedure in its 180‐year history. Effect sizes and confidence intervals, which can be calculated for any data used to calculate p values, provide more and better information about tested hypotheses than p values and NHST.     

生态学假说试验验证的原假说困境

试验方法是生态学假说的主要验证方法之一,但也存在由原假说引发的质疑.Quinn和Dunham(1983)通过对Platt(1964)的假说-演绎模型进行分析,主张生态学不可能存在可以严格被试验验证的原假说.Fisher的证伪主义与Neyman-Pearson(N-P)的非判决性使得统计学原假说不能被严格验证;而生态过程中存在的不同于经典物理学的原假说H₀(α=1,β=0)与不同的备假说H₁′(α′=1,β′=0)的情况,使得生态学原假说也很难得到严格的实验验证.通过降低P值、谨慎选择原假说、对非原假说采取非中心化和双侧验证可分别缓解上述的原假说困境.但统计学的原假说显著性验证(NHST)不应等同于生态学假说中有关因果关系的逻辑证明方法.因此,现有大量基于NHST的生态学假说的方法研究和试验验证的结果与结论都不是绝对的逻辑可靠的.     

P > .05: The incorrect interpretation of “not significant” results is a significant problem

Statistically nonsignificant (p > .05) results from a null hypothesis significance test (NHST) are often mistakenly interpreted as evidence that the null hypothesis is true—that there is “no effect” or “no difference.” However, many of these results occur because the study had low statistical power to detect an effect. Power below 50% is common, in which case a result of no statistical significance is more likely to be incorrect than correct. The inference of “no effect” is not valid even if power is high. NHST assumes that the null hypothesis is true; p is the probability of the data under the assumption that there is no effect. A statistical test cannot confirm what it assumes. These incorrect statistical inferences could be eliminated if decisions based on p values were replaced by a biological evaluation of effect sizes and their confidence intervals. For a single study, the observed effect size is the best estimate of the population effect size, regardless of the p value. Unlike p values, confidence intervals provide information about the precision of the observed effect. In the biomedical and pharmacology literature, methods have been developed to evaluate whether effects are “equivalent,” rather than zero, as tested with NHST. These methods could be used by biological anthropologists to evaluate the presence or absence of meaningful biological effects. Most of what appears to be known about no difference or no effect between sexes, between populations, between treatments, and other circumstances in the biological anthropology literature is based on invalid statistical inference.     

Persistent Controversy in Statistical Approaches in Wildlife Sciences: A Perspective of Students

ABSTRACT The controversy over the use of null hypothesis statistical testing (NHST) has persisted for decades, yet NHST remains the most widely used statistical approach in wildlife sciences and ecology. A disconnect exists between those opposing NHST and many wildlife scientists and ecologists who conduct and publish research. This disconnect causes confusion and frustration on the part of students. We, as students, offer our perspective on how this issue may be addressed. Our objective is to encourage academic institutions and advisors of undergraduate and graduate students to introduce students to various statistical approaches so we can make well-informed decisions on the appropriate use of statistical tools in wildlife and ecological research projects. We propose an academic course that introduces students to various statistical approaches (e.g., Bayesian, frequentist, Fisherian, information theory) to build a foundation for critical thinking in applying statistics. We encourage academic advisors to become familiar with the statistical approaches available to wildlife scientists and ecologists and thus decrease bias towards one approach. Null hypothesis statistical testing is likely to persist as the most common statistical analysis tool in wildlife science until academic institutions and student advisors change their approach and emphasize a wider range of statistical methods.     

Reverend Paley's naturalist revival

This paper analyzes the remarkable popularity of William Paley's argument from design among contemporary naturalists in biology and the philosophy of science. In philosophy of science Elliott Sober has argued that creationism should be excluded from the schools not because it is not science but because it is 'less likely' than evolution according to fairly standard confirmation theory. Creationism is said to have been a plausible scientific option as presented by Paley but no longer to be acceptable according to the same standards that once approved it. In biology C. G. Williams and Richard Dawkins have seen in Paley a proto-adaptationist. This paper shows that the historical assumptions of Sober's arguments are wrong and that the philosophical arguments themselves take alternatives to science to be alternatives in science and conflate the null hypothesis, chance, with a competing explanatory hypothesis. It is also shown that the similarity of Paley's adaptationism to that of contemporary biology is not what it is made out to be.     

Multifactorial analysis of family data ascertained through truncation: a comparative evaluation of two methods of statistical inference.

When family data are ascertained through single selection based on truncation, a prevailing method of analysis is to condition the likelihood function on the proband's actual phenotypic value. An alternative method conditions the likelihood function on the event that the proband's measurement lies in the truncation region. Both methods are contrasted here by using Monte Carlo simulations; identical sets of data were analyzed using both methods. The results suggest that, under either method, (1) parameter estimates are nearly unbiased and (2) likelihood-ratio tests of null hypotheses are approximately distributed as chi 2. However, conditioning on the proband's actual phenotypic value yields considerably less efficient estimates and reduced power for hypothesis tests. A corresponding result also holds under complete ascertainment. It is argued, therefore, that whenever sufficient information is available on the nature of truncation, the alternative approach should be used.     

Using phylogeographic analyses of gene trees to test species status and processes

A gene tree is an evolutionary reconstruction of the genealogical history of the genetic variation found in a sample of homologous genes or DNA regions that have experienced little or no recombination. Gene trees have the potential of straddling the interface between intra- and interspecific evolution. It is precisely at this interface that the process of speciation occurs, and gene trees can therefore be used as a powerful tool to probe this interface. One application is to infer species status. The cohesion species is defined as an evolutionary lineage or set of lineages with genetic exchangeability and/or ecological interchangeability. This species concept can be phrased in terms of null hypotheses that can be tested rigorously and objectively by using gene trees. First, an overlay of geography upon the gene tree is used to test the null hypothesis that the sample is from a single evolutionary lineage. This phase of testing can indicate that the sampled organisms are indeed from a single lineage and therefore a single cohesion species. In other cases, this null hypothesis is not rejected due to a lack of power or inadequate sampling. Alternatively, this null hypothesis can be rejected because two or more lineages are in the sample. The test can identify lineages even when hybridization and lineage sorting occur. Only when this null hypothesis is rejected is there the potential for more than one cohesion species. Although all cohesion species are evolutionary lineages, not all evolutionary lineages are cohesion species. Therefore, if the first null hypothesis is rejected, a second null hypothesis is tested that all lineages are genetically exchangeable and/or ecologically interchangeable. This second test is accomplished by direct contrasts of previously identified lineages or by overlaying reproductive and/or ecological data upon the gene tree and testing for significant transitions that are concordant with the previously identified lineages. Only when this second null hypothesis is rejected is a lineage elevated to the status of cohesion species. By using gene trees in this manner, species can be identified with objective, a priori criteria with an inference procedure that automatically yields much insight into the process of speciation. When one or more of the null hypotheses cannot be rejected, this procedure also provides specific guidance for future work that will be needed to judge species status.     

Near complete loss of retinal ganglion cells in the math5/brn3b double knockout elicits severe reductions of other cell types during retinal development

Retinal ganglion cells (RGCs) are the first cell type to differentiate during retinal histogenesis. It has been postulated that specified RGCs subsequently influence the number and fate of the remaining progenitors to produce the rest of the retinal cell types. However, several genetic knockout models have argued against this developmental role for RGCs. Although it is known that RGCs secrete cellular factors implicated in cell proliferation, survival, and differentiation, until now, limited publications have shown that reductions in the RGC number cause significant changes in these processes. In this study, we observed that Math5 and Brn3b double null mice exhibited over a 99% reduction in the number of RGCs during development. This severe reduction of RGCs is accompanied by a drastic loss in the number of all other retinal cell types that was never seen before. Unlike Brn3b null or Math5 null animals, mice null for both alleles lack an optic nerve and have severe retinal dysfunction. Results of this study support the hypothesis that RGCs play a pivotal role in the late phase of mammalian retina development.     

When Being “Most Likely” Is Not Enough: Examining the Performance of Three Uses of the Parametric Bootstrap in Phylogenetics

Abstract I show that three parametric-bootstrap (PB) applications that have been proposed for phylogenetic analysis, can be misleading as currently implemented. First, I show that simulating a topology estimated from preliminary data in order to determine the sequence length that should allow the best tree obtained from more extensive data to be correct with a desired probability, delivers an accurate estimate of this length only in topological situations in which most preliminary trees are expected to be both correct and statistically significant, i.e. when no further analysis would be needed. Otherwise, one obtains strong underestimates of the length or similarly biased values for incorrect trees. Second, I show that PB-based topology tests that use as null hypothesis the most likely tree congruent with a pre-specified topological relationship alternative to the unconstrained most likely tree, and simulate this tree for P value estimation, produce excessive type I error (from 50% to 600% and higher) when they are applied to null data generated by star-shaped or dichotomous four-taxon topologies. Simulating the most likely star topology for P value estimation results instead in correct type-I-error production even when the null data are generated by a dichotomous topology. This is a strong indication that the star topology is the correct default null hypothesis for phylogenies. Third, I show that PB-estimated confidence intervals (CIs) for the length of a tree branch are generally accurate, although in some situations they can be strongly over- or under-estimated relative to the “true” CI. Attempts to identify a biased CI through a further round of simulations were unsuccessful. Tracing the origin and propagation of parameter estimate error through the CI estimation exercise, showed that the sparseness of site-patterns which are crucial to the estimation of pivotal parameters, can allow homoplasy to bias these estimates and ultimately the PB-based CI estimation. Concluding, I stress that statistical techniques that simulate models estimated from limited data need to be carefully calibrated, and I defend the point that pattern-sparseness assessment will be the next frontier in the statistical analysis of phylogenies, an effort that will require taking advantage of the merits of black-box maximum-likelihood approaches and of insights from intuitive, site-pattern-oriented approaches like parsimony.     

The value of being a resource specialist: behavioral support for a neural hypothesis

The neural hypothesis of diet breadth proposes that selecting an appropriate behavior is more efficient if simple or exaggerated cues can be used as a basis of decision making rather than making a choice among many complex sensory inputs. I propose that simple signals overcome the problem of multiple sensory inputs and the consequent need for the brain to decide among inputs from these multiple channels. Experiments on grasshoppers show that there is a significant time cost in having to make a choice, relative to situations in which individuals have grown accustomed to having no choice. Those with a choice were shown to be less decisive by two different measures than those without a choice. It is argued that the data, showing lengthy decision times as a result of having a choice, would involve a significant ecological risk. It is further argued that the reduced risk of quick decisions would favor specialization of resource use. The evolution of resource-specific cues that have often been called sign stimuli are considered critical elements of restricted resource use.     

The genetic hypothesis for susceptibility to lepromatous leprosy

Summary Evidence for genetic influence of the host response to infection with Mycobacterium leprae is reviewed. A complex segregation analysis is performed on data for 91 families from Mactan, Philippines, in each of which at least one offspring developed lepromatous leprosy. The data are not found to be inconsistent with an autosomal recessive hypothesis for susceptibility to lepromatous leprosy. Heritability estimates in the range of 80% were calculated for sib-sib pairs under the multifactorial hypothesis for susceptibility. It is argued that the multifactorial hypothesis is more in keeping with available immunologic, epidemiologic, and demographic data than is the single gene hypothesis.     

A Bayesian analysis of the two-period crossover design for clinical trials

Statisticians have been critical of the use of the two-period crossover designs for clinical trials because the estimate of the treatment difference is biased when the carryover effects of the two treatments are not equal. In the standard approach, if the null hypothesis of equal carryover effects is not rejected, data from both periods are used to estimate and test for treatment differences; if the null hypothesis is rejected, data from the first period alone are used. A Bayesian analysis based on the Bayes factor against unequal carryover effects is given. Although this Bayesian approach avoids the "all-or-nothing" decision inherent in the standard approach, it recognizes that with small trials it is difficult to provide unequivocal evidence that the carryover effects of the two treatments are equal, and thus that the interpretation of the difference between treatment effects is highly dependent on a subjective assessment of the reality or not of equal carryover effects.     

Gadd45 mutations are uncommon in human tumour cell lines

GADD45 is an evolutionarily conserved gene that encodes a small acidic, nuclear protein and is an example of a p53 responsive gene. Gadd45 protein has been shown to interact with PCNA and also p21^waf1. It has been implicated in growth arrest, DNA repair, chromatin structure and signal transduction. The confusing biochemical data has been clarified by the demonstration that Gadd45 null mice have a phenotype strikingly similar to that of p53 null mice, being tumour prone and showing marked genomic instability. We have tested the hypothesis that mutations in the GADD45 coding region might substitute for p53 abnormalities in tumour cell lines where p53 is wild type. After generating cDNA from mRNA in a panel of 24 cell lines we sequenced the GADD45 cDNA and have demonstrated that no mutations can be observed, even in the p53 wild type cell lines. Such data suggest that Gadd45 mutations are uncommon in human cancer. From this we postulate that, despite the phenotype of the GADD45 null mouse, GADD45 is unlikely to be the key mechanistic determinant of the tumour suppressor activity of the p53 pathway.
Note on nomenclature: We have employed GADD45 to designate the gene and Gadd45 to designate the encoded protein. This gene has also be denoted GADD45 α elsewhere in the literature.     

Coloured noise or low-dimensional chaos?

Devising a method capable of distinguishing a low-dimensional chaotic signal that might be embedded in a noisy stochastic process has become a major challenge for those involved in time-series analysis. Here a null hypothesis approach is used in conjunction with a known nonlinear predictive test, to probe for the presence of chaos in epidemiological data. A probabilistic set of rules is used to stimulate a historic record of New York City measles outbreaks, generally understood to be governed by a chaotic attractor. The simulated runs of 'surrogate data' are carefully constructed so as to be free from any underlying low-dimensional chaotic process. They therefore serve as a useful null model against which to test the observed time series. However, despite the assumed differences between the dynamics of measles outbreaks and the null model, a nonlinear predictive scheme is found to be unable to differentiate between their characteristic time series. The methodology confirms that, if there is in fact a chaotic signal in the measles data, it is extremely difficult to detect in time series of such limited length. The results have general relevance to the analysis of physical, ecological and environmental time series.     

Use of Ranks in Comparisons of Variability in the Non-Normal Populations

A method of analysis for comparing the variability of two samples drawn from two populations has been developed. The method is also suitable for the nonnumeric form of data. A test based on ordered observations for testing the null hypothesis of equality of two variances has been given. The test statistic is a function of the sum of ranks assigned to smaller size sample. Ranking procedure has been modified to depict the variability in the data by the sum of ranks. The null distribution of the test-statistic has been worked out for small samples and it turns out to be chi-square distribution for large samples. The analytical procedure has been explained by a numerical example on the productivity and production of rice and wheat in India from 1950–51 to 1983–84.     

MPowering ecologists: community assembly tools for community assembly rules

Null model tests of presence–absence data (‘NMTPAs’) provide important tools for inferring effects of competition, facilitation, habitat filtering, and other ecological processes from observational data. Many NMTPAs have been developed, but they often yield conflicting conclusions when applied to the same data. Type I and II error rates, size, power, robustness and bias provide important criteria for assessing which tests are valid, but these criteria need to be evaluated contingent on the sample size, null hypothesis of interest, and assumptions that are appropriate for the data set that is being analyzed. In this paper, we confirm that this is the case using the software MPower, evaluating the validity of NMTPAs contingent on the null hypothesis being tested, assumptions that can be made, and sample size. Evaluating the validity of NMTPAs contingent on these factors is important towards ensuring that reliable inferences are drawn from observational data about the processes controlling community assembly.     

Biological implication for loss of function at major histocompatibility complex loci

Despite relatively frequent gene or segment duplications, the number of functional loci in the major histocompatibility complex (MHC) is relatively small. The dual function of MHC molecules (triggering the immune system and limiting T-cell receptor repertoires) is likely to balance the number of functional loci. The effect of this dual function on the number of functional MHC loci has been argued mainly in the theoretical and computer simulation studies, but the evidence from empirical data has not been fully examined. Here, we attempt to evaluate this effect based on the analysis of nucleotide sequence data. We hypothesized that due to the dual function, even becoming a pseudogene (pseudogenization) of MHC is advantageous for the organisms. To evaluate this hypothesis, we compared the distribution of the waiting time (T (W)) till pseudogenization for HLA (human MHC) with that of the human olfactory receptor (OR) and bitter taste receptor (T2R) genes. The result shows that T (W) in HLA has a tendency to be relatively shorter as the emergence time (T) of the gene becomes older, while in OR T (W) becomes proportionally longer as T becomes older and in T2R it is almost null irrespective of T. Furthermore, T (W) in HLA is strongly influenced by the extent of functional differentiation in the peptide-binding region. Taken together, these results show that MHC molecules have optimal numbers of functional loci, and these numbers are regulated by the advantageous pseudogenization of duplicated copies.     

Stochasticity,spikes and decoding: sufficiency and utility of order statistics

For over 75 years it has been clear that the number of spikes in a neural response is an important part of the neuronal code. Starting as early as the 1950’s with MacKay and McCullough, there has been speculation over whether each spike and its exact time of occurrence carry information. Although it is obvious that the firing rate carries information it has been less clear as to whether there is information in exactly timed patterns, when they arise from the dynamics of the neurons and networks, as opposed to when they represent some strong external drive that entrains them. One strong null hypothesis that can be applied is that spike trains arise from stochastic sampling of an underlying deterministic temporally modulated rate function, that is, there is a time-varying rate function. In this view, order statistics seem to provide a sufficient theoretical construct to both generate simulated spike trains that are indistinguishable from those observed experimentally, and to evaluate (decode) the data recovered from experiments. It remains to learn whether there are physiologically important signals that are not described by such a null hypothesis. This article is part of a special issue on Neuronal Dynamics of Sensory Coding.     

Selection of evolutionary models for phylogenetic hypothesis testing using parametric methods

Recent molecular studies have incorporated the parametric bootstrap method to test a priori hypotheses when the results of molecular based phylogenies are in conflict with these hypotheses. The parametric bootstrap requires the specification of a particular substitutional model, the parameters of which will be used to generate simulated, replicate DNA sequence data sets. It has been both suggested that, (a) the method appears robust to changes in the model of evolution, and alternatively that, (b) as realistic model of DNA substitution as possible should be used to avoid false rejection of a null hypothesis. Here we empirically evaluate the effect of suboptimal substitution models when testing hypotheses of monophyly with the parametric bootstrap using data sets of mtDNA cytochrome oxidase I and II (COI and COII) sequences for Macaronesian Calathus beetles, and mitochondrial 16S rDNA and nuclear ITS2 sequences for European Timarcha beetles. Whether a particular hypothesis of monophyly is rejected or accepted appears to be highly dependent on whether the nucleotide substitution model being used is optimal. It appears that a parameter rich model is either equally or less likely to reject a hypothesis of monophyly where the optimal model is unknown. A comparison of the performance of the Kishino–Hasegawa (KH) test shows it is not as severely affected by the use of suboptimal models, and overall it appears to be a less conservative method with a higher rate of failure to reject null hypotheses.     

Least and most powerful phylogenetic tests to elucidate the origin of the seed plants in the presence of conflicting signals under misspecified models

Several tests of molecular phylogenies have been proposed over the last decades, but most of them lead to strikingly different P-values. I propose that such discrepancies are principally due to different forms of null hypotheses. To support this hypothesis, two new tests are described. Both consider the composite null hypothesis that all the topologies are equidistant from the true but unknown topology. This composite hypothesis can either be reduced to the simple hypothesis at the least favorable distribution (frequentist significance test [FST]) or to the maximum likelihood topology (frequentist hypothesis test [FHT]). In both cases, the reduced null hypothesis is tested against each topology included in the analysis. The tests proposed have an information-theoretic justification, and the distribution of their test statistic is estimated by a nonparametric bootstrap, adjusting P-values for multiple comparisons. I applied the new tests to the reanalysis of two chloroplast genes, psaA and psbB, and compared the results with those of previously described tests. As expected, the FST and the FHT behaved approximately like the Shimodaira-Hasegawa test and the bootstrap, respectively. Although the tests give overconfidence in a wrong tree when an overly simple nucleotide substitution model is assumed, more complex models incorporating heterogeneity among codon positions resolve some conflicts. To further investigate the influence of the null hypothesis, a power study was conducted. Simulations showed that FST and the Shimodaira-Hasegawa test are the least powerful and FHT is the most powerful across the parameter space. Although the size of all the tests is affected by misspecification, the two new tests appear more robust against misspecification of the model of evolution and consistently supported the hypothesis that the Gnetales are nested within gymnosperms.