Assessing the goodness of fit of logistic regression models in large samples: A modification of the Hosmer-Lemeshow test

Evaluating the goodness of fit of logistic regression models is crucial to ensure the accuracy of the estimated probabilities. Unfortunately, such evaluation is problematic in large samples. Because the power of traditional goodness of fit tests increases with the sample size, practically irrelevant discrepancies between estimated and true probabilities are increasingly likely to cause the rejection of the hypothesis of perfect fit in larger and larger samples. This phenomenon has been widely documented for popular goodness of fit tests, such as the Hosmer-Lemeshow test. To address this limitation, we propose a modification of the Hosmer-Lemeshow approach. By standardizing the noncentrality parameter that characterizes the alternative distribution of the Hosmer-Lemeshow statistic, we introduce a parameter that measures the goodness of fit of a model but does not depend on the sample size. We provide the methodology to estimate this parameter and construct confidence intervals for it. Finally, we propose a formal statistical test to rigorously assess whether the fit of a model, albeit not perfect, is acceptable for practical purposes. The proposed method is compared in a simulation study with a competing modification of the Hosmer-Lemeshow test, based on repeated subsampling. We provide a step-by-step illustration of our method using a model for postneonatal mortality developed in a large cohort of more than 300 000 observations.     

A goodness-of-fit test for multinomial logistic regression

This article presents a score test to check the fit of a logistic regression model with two or more outcome categories. The null hypothesis that the model fits well is tested against the alternative that residuals of samples close to each other in covariate space tend to deviate from the model in the same direction. We propose a test statistic that is a sum of squared smoothed residuals, and show that it can be interpreted as a score test in a random effects model. By specifying the distance metric in covariate space, users can choose the alternative against which the test is directed, making it either an omnibus goodness-of-fit test or a test for lack of fit of specific model variables or outcome categories.     

Goodness of Fit Tests with Misclassified Data Based on φ‐Divergences

The well known χ² goodness of fit test for a multinomial distribution is generally biased when observations are subject to misclassification. In this paper, based on a double sampling scheme, the family of φ‐divergence test statistics is introduced for testing goodness of fit under misclassification of the data. The case of binomial data is discussed and an illustrative example is also given.     

Fit and Frailties in Proportional Hazards Regression

We exploit a conjectured equivalence between proportional hazards models with frailties and a particular subclass of non proportional hazards models, specifically those with declining effects, to address the question of fit. A goodness of fit test of the proportional hazards assumption against an alternative of declining regression effect is equivalent to a test for the presence of frailties. Such tests are now widely available in standard software. Although a number of tests of the proportional hazards assumption have been developed there is no test that directly formulates the alternative in terms of a non‐specified monotonic decline in regression effect and that enables a quantification of this in terms of a simple index. The index we obtain lies between zero and one such that, for any given set of covariates, values of the index close to one indicate that the fit cannot essentially be improved by allowing the possibility of regression effects to decline. Values closer to zero and away from one indicate that the fit can be improved by relaxing the proportional hazards constraint in this particular direction. (© 2004 WILEY‐VCH Verlag GmbH & Co. KGaA, Weinheim)     

时间序列数据记忆性的判别方法

目的：发展时间序列数据记忆的判别方法并用于离子通道数据记忆性的判别研究。方法：应用Pearson检验与曲线拟合的方法,识别数据的记忆性。结果：得到数据记忆性的判别方法,讨论了离子通道记忆性。结论：Pearson检验与曲线拟合法简单易用,离子通道具有记忆性。     

Modeling compositional heterogeneity

Compositional heterogeneity among lineages can compromise phylogenetic analyses, because models in common use assume compositionally homogeneous data. Models that can accommodate compositional heterogeneity with few extra parameters are described here, and used in two examples where the true tree is known with confidence. It is shown using likelihood ratio tests that adequate modeling of compositional heterogeneity can be achieved with few composition parameters, that the data may not need to be modelled with separate composition parameters for each branch in the tree. Tree searching and placement of composition vectors on the tree are done in a Bayesian framework using Markov chain Monte Carlo (MCMC) methods. Assessment of fit of the model to the data is made in both maximum likelihood (ML) and Bayesian frameworks. In an ML framework, overall model fit is assessed using the Goldman-Cox test, and the fit of the composition implied by a (possibly heterogeneous) model to the composition of the data is assessed using a novel tree-and model-based composition fit test. In a Bayesian framework, overall model fit and composition fit are assessed using posterior predictive simulation. It is shown that when composition is not accommodated, then the model does not fit, and incorrect trees are found; but when composition is accommodated, the model then fits, and the known correct phylogenies are obtained.     

A note on goodness of fit of a population to Hardy-Weinberg structure.

The test of goodness of fit of a population to Hardy-Weinberg equilibrium given by Levene [2] and Haldane [1] is valid within a widely accepted recipe for testing goodness of fit of a composite hypothesis. The nature of the result of Cannings and Edwards [3] is described. The result was shown to be quite different than they claimed and, although possible of some interest, not relevant to the testing of goodness of fit to Hardy-Weinberg structure.     

A goodness‐of‐fit test for occupancy models with correlated within‐season revisits

Occupancy modeling is important for exploring species distribution patterns and for conservation monitoring. Within this framework, explicit attention is given to species detection probabilities estimated from replicate surveys to sample units. A central assumption is that replicate surveys are independent Bernoulli trials, but this assumption becomes untenable when ecologists serially deploy remote cameras and acoustic recording devices over days and weeks to survey rare and elusive animals. Proposed solutions involve modifying the detection‐level component of the model (e.g., first‐order Markov covariate). Evaluating whether a model sufficiently accounts for correlation is imperative, but clear guidance for practitioners is lacking. Currently, an omnibus goodness‐of‐fit test using a chi‐square discrepancy measure on unique detection histories is available for occupancy models (MacKenzie and Bailey, Journal of Agricultural, Biological, and Environmental Statistics, 9, 2004, 300; hereafter, MacKenzie–Bailey test). We propose a join count summary measure adapted from spatial statistics to directly assess correlation after fitting a model. We motivate our work with a dataset of multinight bat call recordings from a pilot study for the North American Bat Monitoring Program. We found in simulations that our join count test was more reliable than the MacKenzie–Bailey test for detecting inadequacy of a model that assumed independence, particularly when serial correlation was low to moderate. A model that included a Markov‐structured detection‐level covariate produced unbiased occupancy estimates except in the presence of strong serial correlation and a revisit design consisting only of temporal replicates. When applied to two common bat species, our approach illustrates that sophisticated models do not guarantee adequate fit to real data, underscoring the importance of model assessment. Our join count test provides a widely applicable goodness‐of‐fit test and specifically evaluates occupancy model lack of fit related to correlation among detections within a sample unit. Our diagnostic tool is available for practitioners that serially deploy survey equipment as a way to achieve cost savings.     

Modified GEE and Goodness of the Marginal Fit (GOMF) Test with Correlated Binary Responses for Contingency Tables

This paper addresses testing the goodness of fit of models for marginal probabilities estimated by generalized estimating equations. We develop a modified version of generalized estimating equation and a goodness‐of‐fit test based on the fitted marginal means. The test statistic is easy to compute and has a simple reference distribution. Its performance is evaluated asymptotically and in small samples. It is also compared to the deviance and Pearson X² statistics. Example applications are given. (© 2004 WILEY‐VCH Verlag GmbH & Co. KGaA, Weinheim)     

Testing goodness-of-fit in logistic case-control studies

We present a goodness-of-fit test for the logistic regressionmodel under case-control sampling. The test statistic is constructedvia a discrepancy between two competing kernel density estimatorsof the underlying conditional distributions given case-controlstatus. The proposed goodness-of-fit test is shown to comparevery favourably with previously proposed tests for case-controlsampling in terms of power. The test statistic can be easilycomputed as a quadratic form in the residuals from a prospectivelogistic regression maximum likelihood fit. In addition, theproposed test is affine invariant and has an alternative representationin terms of empirical characteristic functions.     

Testing goodness of fit for stochastic models of carcinogenesis.

This paper considers the utility of statistical goodness of fit testing in the context of mechanistic models of carcinogenesis. Two stochastic models of carcinogenesis were tested with several sets of experimental and epidemiological data using a formal goodness of fit test specially designed to accommodate censored observations: these were the two-stage model allowing for clonal expansion of initiated cells and its simpler version with gamma distributed promotion time. The results of this application, supplemented by visual examination of local likelihood kernel estimates of the hazard function and the corresponding model-based estimates, show that mechanistic models of carcinogenesis provide a good fit to the data in the majority of cases under study.     

Testing the fit of a quantal model of neurotransmission

Many studies of synaptic transmission have assumed a parametric model to estimate the mean quantal content and size or the effect upon them of manipulations such as the induction of long-term potentiation. Classical tests of fit usually assume that model parameters have been selected independently of the data. Therefore, their use is problematic after parameters have been estimated. We hypothesized that Monte Carlo (MC) simulations of a quantal model could provide a table of parameter-independent critical values with which to test the fit after parameter estimation, emulating Lilliefors's tests. However, when we tested this hypothesis within a conventional quantal model, the empirical distributions of two conventional goodness-of-fit statistics were affected by the values of the quantal parameters, falsifying the hypothesis. Notably, the tests' critical values increased when the combined variances of the noise and quantal-size distributions were reduced, increasing the distinctness of quantal peaks. Our results support two conclusions. First, tests that use a predetermined critical value to assess the fit of a quantal model after parameter estimation may operate at a differing unknown level of significance for each experiment. Second, a MC test enables a valid assessment of the fit of a quantal model after parameter estimation.     

Growth and energy allocation in the deep-sea urchin Echinus affinis

Growth of regular echinoids, expressed as test diameter through time, generally shows a sigmoidal pattern. However, when urchin size is considered in terms of test volume we show that growth of the deep-sea echinoid Echinus affinis is ultimately linear, rather than saturating. We construct a simple allometric model of energy allocation that produces linear growth in volume in mature urchins by allocating an increasing proportion of net assimilate to reproduction. This model provides an excellent fit to the observed growth curve data. Data on gonad weight as a function of test diameter allow us to test the relationship between allocation to reproduction and urchin size predicted from the growth curve fit. Simultaneous fitting of the growth curve and gonad weight data allow us to consider a model where the allometry of net assimilation is allowed to vary. We investigate possible explanations for the fact that net assimilation appears to rise faster than linearly with weight in E. affinis. We conclude that strategic models of individual energetics provide a useful tool for the analysis of the limited data available on deep-sea populations.     

A smooth test of goodness-of-fit for growth curves and monotonic nonlinear regression models

We propose a goodness-of-fit test for growth curves based on an adaptation of the data-driven smooth test paradigm. It is simple to apply and can assess the fit of a model to a set of growth experiences. A simulation study shows that for small samples, the test holds its level. Moreover, its power is found to be generally greater than existing tests. The article concludes by revisiting the long-standing problem of validating a model for the growth of human stature.     

S-Sample Smooth Goodness of Fit Testing: Rederivation and Monte Carlo Assessment

S-sample smooth goodness of fit tests may be constructed using components from one sample goodness of fit testing. Each sample could be assessed for consistency with a target distribution using these components, although that is not our objective here. Contrasts in the components may be used to assess consistency of the samples with each other. If all the samples are consistent, we could then conveniently perform a one-sample goodness of fit test for the target distribution. If the samples are not consistent, an LSD-type analysis can be performed on the one-sample components to identify where the differences between occur. This approach gives a detailed and informative scrutiny of the data.     

Use of the Score Statistic for Testing Goodness-of-Fit of Some Generalised Distributions

The use of the score statistic to test whether a generalised distribution gives an improved fit over a non-generalised distribution is recommended. The score statistic for a generalised exponential family is derived. Several specific examples are given.     

Cryo‐EM model validation using independent map reconstructions

An increasing number of cryo‐electron microscopy (cryo‐EM) density maps are being generated with suitable resolution to trace the protein backbone and guide sidechain placement. Generating and evaluating atomic models based on such maps would be greatly facilitated by independent validation metrics for assessing the fit of the models to the data. We describe such a metric based on the fit of atomic models with independent test maps from single particle reconstructions not used in model refinement. The metric provides a means to determine the proper balance between the fit to the density and model energy and stereochemistry during refinement, and is likely to be useful in determining values of model building and refinement metaparameters quite generally.     

Goodness of Fit for the Geometric Distribution

A Neyman-type smooth test of goodness of fit is derived for the geometric distribution. Some small-sample critical points are given, with two examples.     

Predicting Performance in Higher Education Using Proximal Predictors

We studied the validity of two methods for predicting academic performance and student-program fit that were proximal to important study criteria. Applicants to an undergraduate psychology program participated in a selection procedure containing a trial-studying test based on a work sample approach, and specific skills tests in English and math. Test scores were used to predict academic achievement and progress after the first year, achievement in specific course types, enrollment, and dropout after the first year. All tests showed positive significant correlations with the criteria. The trial-studying test was consistently the best predictor in the admission procedure. We found no significant differences between the predictive validity of the trial-studying test and prior educational performance, and substantial shared explained variance between the two predictors. Only applicants with lower trial-studying scores were significantly less likely to enroll in the program. In conclusion, the trial-studying test yielded predictive validities similar to that of prior educational performance and possibly enabled self-selection. In admissions aimed at student-program fit, or in admissions in which past educational performance is difficult to use, a trial-studying test is a good instrument to predict academic performance.