Distribution-free ROC analysis using binary regression techniques

Receiver operating characteristic (ROC) regression methodology is used to identify factors that affect the accuracy of medical diagnostic tests. In this paper, we consider a ROC model for which the ROC curve is a parametric function of covariates but distributions of the diagnostic test results are not specified. Covariates can be either common to all subjects or specific to those with disease. We propose a new estimation procedure based on binary indicators defined by the test result for a diseased subject exceeding various specified quantiles of the distribution of test results from non-diseased subjects with the same covariate values. This procedure is conceptually and computationally simplified relative to existing procedures. Simulation study results indicate that the approach has fairly high statistical efficiency. The new ROC regression methodology is used to evaluate childhood measurements of body mass index as a predictive marker of adult obesity.     

Application of the time-dependent ROC curves for prognostic accuracy with multiple biomarkers

The rapid advancement in molecule technology has led to the discovery of many markers that have potential applications in disease diagnosis and prognosis. In a prospective cohort study, information on a panel of biomarkers as well as the disease status for a patient are routinely collected over time. Such information is useful to predict patients' prognosis and select patients for targeted therapy. In this article, we develop procedures for constructing a composite test with optimal discrimination power when there are multiple markers available to assist in prediction and characterize the accuracy of the resulting test by extending the time-dependent receiver operating characteristic (ROC) curve methodology. We employ a modified logistic regression model to derive optimal linear composite scores such that their corresponding ROC curves are maximized at every false positive rate. We provide theoretical justification for using such a model for prognostic accuracy. The proposed method allows for time-varying marker effects and accommodates censored failure time outcome. When the effects of markers are approximately constant over time, we propose a more efficient estimating procedure under such models. We conduct numerical studies to evaluate the performance of the proposed procedures. Our results indicate the proposed methods are both flexible and efficient. We contrast these methods with an application concerning the prognostic accuracies of expression levels of six genes.     

Downscaling European species atlas distributions to a finer resolution: implications for conservation planning

Aim One of the limitations to using species’ distribution atlases in conservation planning is their coarse resolution relative to the needs of local planners. In this study, a simple approach to downscale original species atlas distributions to a finer resolution is outlined. If such a procedure yielded accurate downscaled predictions, then it could be an aid to using available distribution atlases in real‐world local conservation decisions. Location Europe. Methods An iterative procedure based on generalized additive modelling is used to downscale original European 50 × 50 km distributions of 2189 plant and terrestrial vertebrate species to c. 10 × 10 km grid resolution. Models are trained on 70% of the original data and evaluated on the remaining 30%, using the receiver operating characteristic (ROC) procedure. Fitted models are then interpolated to a finer resolution. A British dataset comprising distributions of 81 passerine‐bird species in a 10 × 10 km grid is used as a test bed to assess the accuracy of the downscaled predictions. European‐wide, downscaled predictions are further evaluated in terms of their ability to reproduce: (1) spatial patterns of coincidence in species richness scores among different groups; and (2) spatial patterns of coincidence in richness, rarity and complementarity hotspots. Results There was a generally good agreement between downscaled and observed fine‐resolution distributions for passerine species in Britain (median Jaccard similarity = 70%; lower quartile = 36%; upper quartile = 88%). In contrast, the correlation between downscaled and observed passerine species richness was relatively low (rho = 0.31) indicating a pattern of error propagation through the process of overlaying downscaled distributions for many species. It was also found that measures of model accuracy in fitting original data (ROC) were a poor predictor of models’ ability to interpolate distributions at fine resolutions (rho = ?0.10). Although European hotspots were not fully coincident between observed and modelled coarse‐resolution data, or between modelled coarse resolution and modelled downscaled data, there was evidence that downscaled distributions were able to maintain original cross‐taxon coincidence of species‐richness scores, at least for terrestrial vertebrate groups. Downscaled distributions were also able to uncover important environmental gradients otherwise blurred by coarse‐resolution data. Main conclusions Despite uncertainties, downscaling procedures may prove useful to identify reserves that are more meaningfully related to local patterns of environmental variation. Potential errors arising from the presence of false positives may be reduced if downscaled‐distribution records projected to occur outside the range of original coarse‐resolution data are excluded. However, the usefulness of this procedure may be limited to data‐rich regions. If downscaling procedures are applied to data‐poor regions, then there is a need to undertake further research to understand the structure of error in models. In particular, it would be important to investigate which species are poorly modelled, where and why. Without such an assessment it is difficult to support unsupervised use of downscaled data in most real‐world situations.     

我国林火发生预测模型研究进展

通过文献回顾,总结了国内林火发生预测模型的研究现状,并从林火发生驱动因子、林火发生概率预测模型、林火发生频次预测模型和模型检验方法等方面进行归纳分析。得出以下结论: 1)气象、地形、植被、可燃物、人类活动等因素是影响林火发生及模型预测精度的主要驱动因子;2)林火发生概率模型中,地理加权逻辑斯蒂回归模型考虑了变量之间的空间相关性,Gompit回归模型适宜非对称结构的林火数据,随机森林模型不需要多重共线性检验,在避免过度拟合的同时提高了预测精度,是林火发生概率预测模型的优选方法之一;3)林火发生频次模型中,负二项回归模型更适合对过度离散数据进行模拟,零膨胀模型和栅栏模型可以处理林火数据中包含大量零值的问题;4)ROC检验、AIC检验、似然比检验和Wald检验方法是林火概率和频次模型的常用检验方法。林火发生预测模型研究仍是我国当前林火管理工作的重点,预测模型的选择需要依据不同地区林火数据特点。此外,构建林火预测模型时需要考虑更多的影响因素,以提高模型预测精度;未来,需要进一步探索其他数学模型在林火发生预测中的应用,不断提高林火发生预测模型的准确度。     

The Mizon–Richard Encompassing Test for the Cox and Aalen Additive Hazards Models

Summary .   The Cox hazards model ( Cox, 1972 , Journal of the Royal Statistical Society, Series B 34, 187–220) for survival data is routinely used in many applied fields, sometimes, however, with too little emphasis on the fit of the model. A useful alternative to the Cox model is the Aalen additive hazards model ( Aalen, 1980 , in Lecture Notes in Statistics-2 , 1–25) that can easily accommodate time changing covariate effects. It is of interest to decide which of the two models that are most appropriate to apply in a given application. This is a nontrivial problem as these two classes of models are nonnested except only for special cases. In this article we explore the Mizon–Richard encompassing test for this particular problem. It turns out that it corresponds to fitting of the Aalen model to the martingale residuals obtained from the Cox regression analysis. We also consider a variant of this method, which relates to the proportional excess model ( Martinussen and Scheike, 2002 , Biometrika 89, 283–298). Large sample properties of the suggested methods under the two rival models are derived. The finite-sample properties of the proposed procedures are assessed through a simulation study. The methods are further applied to the well-known primary biliary cirrhosis data set.     

Using cross-validation to evaluate predictive accuracy of survival risk classifiers based on high-dimensional data

Developments in whole genome biotechnology have stimulated statistical focus on prediction methods. We review here methodology for classifying patients into survival risk groups and for using cross-validation to evaluate such classifications. Measures of discrimination for survival risk models include separation of survival curves, time-dependent ROC curves and Harrell's concordance index. For high-dimensional data applications, however, computing these measures as re-substitution statistics on the same data used for model development results in highly biased estimates. Most developments in methodology for survival risk modeling with high-dimensional data have utilized separate test data sets for model evaluation. Cross-validation has sometimes been used for optimization of tuning parameters. In many applications, however, the data available are too limited for effective division into training and test sets and consequently authors have often either reported re-substitution statistics or analyzed their data using binary classification methods in order to utilize familiar cross-validation. In this article we have tried to indicate how to utilize cross-validation for the evaluation of survival risk models; specifically how to compute cross-validated estimates of survival distributions for predicted risk groups and how to compute cross-validated time-dependent ROC curves. We have also discussed evaluation of the statistical significance of a survival risk model and evaluation of whether high-dimensional genomic data adds predictive accuracy to a model based on standard covariates alone.     

Mixed Model Discrete Regression

Models and estimention procedures are given for linear regression models in discrete distributions when the regression contains both fixed and random effects. The methods are developed for discrete variables with typically a small number of possible outcomes such as occurs in ordinal regression. The method is applied to a problem arising in the comparison of microbiological test methods.     

Assessing the goodness of fit of logistic regression models in large samples: A modification of the Hosmer-Lemeshow test

Evaluating the goodness of fit of logistic regression models is crucial to ensure the accuracy of the estimated probabilities. Unfortunately, such evaluation is problematic in large samples. Because the power of traditional goodness of fit tests increases with the sample size, practically irrelevant discrepancies between estimated and true probabilities are increasingly likely to cause the rejection of the hypothesis of perfect fit in larger and larger samples. This phenomenon has been widely documented for popular goodness of fit tests, such as the Hosmer-Lemeshow test. To address this limitation, we propose a modification of the Hosmer-Lemeshow approach. By standardizing the noncentrality parameter that characterizes the alternative distribution of the Hosmer-Lemeshow statistic, we introduce a parameter that measures the goodness of fit of a model but does not depend on the sample size. We provide the methodology to estimate this parameter and construct confidence intervals for it. Finally, we propose a formal statistical test to rigorously assess whether the fit of a model, albeit not perfect, is acceptable for practical purposes. The proposed method is compared in a simulation study with a competing modification of the Hosmer-Lemeshow test, based on repeated subsampling. We provide a step-by-step illustration of our method using a model for postneonatal mortality developed in a large cohort of more than 300 000 observations.     

Partial AUC estimation and regression

Accurate diagnosis of disease is a critical part of health care. New diagnostic and screening tests must be evaluated based on their abilities to discriminate diseased from nondiseased states. The partial area under the receiver operating characteristic (ROC) curve is a measure of diagnostic test accuracy. We present an interpretation of the partial area under the curve (AUC), which gives rise to a nonparametric estimator. This estimator is more robust than existing estimators, which make parametric assumptions. We show that the robustness is gained with only a moderate loss in efficiency. We describe a regression modeling framework for making inference about covariate effects on the partial AUC. Such models can refine knowledge about test accuracy. Model parameters can be estimated using binary regression methods. We use the regression framework to compare two prostate-specific antigen biomarkers and to evaluate the dependence of biomarker accuracy on the time prior to clinical diagnosis of prostate cancer.     

Predicting distributions of seven bitterling fishes in northern Kyushu,Japan

The distributions of seven bitterling species and subspecies—Tanakia lanceolata, T. limbata, Acheilognathus tabira nakamurae, A. rhombeus, Rhodeus ocellatus kurumeus, R. ocellatus ocellatus, and R. atremius atremius—in northern Kyushu were predicted using generalized linear models (GLMs) in order to provide information helpful for conserving native bitterlings and preventing the expansion of alien bitterling species. Predictions were made according to the following procedure: (1) a set of GLMs for each species was formulated using environmental data from 710 sites that were derived using digital maps and GIS software, from which the best fit model for each species was selected using the Akaike information criterion for predicting the fish occurrence, (2) model performance was evaluated based on the receiver-operating characteristics (ROC) analysis using occurrence and environmental data from 362 sites, and (3) potential distributions of the bitterling were analyzed using the best fit models and environmental data for 1,272 sites, of which 200 data points without occurrence data were prepared. The best fit models revealed that 4–6 environmental factors were important in predicting seven bitterling distributions, which was supported by the area under the ROC curve (AUC) values of these fishes ranging from 0.753 to 0.927. The AUC values in model evaluation were significantly greater than 0.5 for six fishes, suggesting the moderate accuracies of these best fit models for predicting the fish distributions. These predictive models can be used for evaluating potential native bitterling richness and the potential distribution expansion of an alien subspecies.     

Adjusting confounders in ranking biomarkers: a model-based ROC approach

High-throughput studies have been extensively conducted in the research of complex human diseases. As a representative example, consider gene-expression studies where thousands of genes are profiled at the same time. An important objective of such studies is to rank the diagnostic accuracy of biomarkers (e.g. gene expressions) for predicting outcome variables while properly adjusting for confounding effects from low-dimensional clinical risk factors and environmental exposures. Existing approaches are often fully based on parametric or semi-parametric models and target evaluating estimation significance as opposed to diagnostic accuracy. Receiver operating characteristic (ROC) approaches can be employed to tackle this problem. However, existing ROC ranking methods focus on biomarkers only and ignore effects of confounders. In this article, we propose a model-based approach which ranks the diagnostic accuracy of biomarkers using ROC measures with a proper adjustment of confounding effects. To this end, three different methods for constructing the underlying regression models are investigated. Simulation study shows that the proposed methods can accurately identify biomarkers with additional diagnostic power beyond confounders. Analysis of two cancer gene-expression studies demonstrates that adjusting for confounders can lead to substantially different rankings of genes.     

ROC analysis with multiple classes and multiple tests: methodology and its application in microarray studies

The accuracy of a single diagnostic test for binary outcome can be summarized by the area under the receiver operating characteristic (ROC) curve. Volume under the surface and hypervolume under the manifold have been proposed as extensions for multiple class diagnosis (Scurfield, 1996, 1998). However, the lack of simple inferential procedures for such measures has limited their practical utility. Part of the difficulty is that calculating such quantities may not be straightforward, even with a single test. The decision rule used to generate the ROC surface requires class probability assessments, which are not provided by the tests. We develop a method based on estimating the probabilities via some procedure, for example, multinomial logistic regression. Bootstrap inferences are proposed to account for variability in estimating the probabilities and perform well in simulations. The ROC measures are compared to the correct classification rate, which depends heavily on class prevalences. An example of tumor classification with microarray data demonstrates that this property may lead to substantially different analyses. The ROC-based analysis yields notable decreases in model complexity over previous analyses.     

Non‐equilibrium in plant distribution models – only an issue for introduced or dispersal limited species?

Species distribution models rely on the assumption that species' distributions are at equilibrium with environmental conditions within a region – i.e. they occur in all suitable habitats. If this assumption holds, species occurrence should be predictable from measures of the environment. Introduced species may be poor candidates for distribution models due to their presumed lack of equilibrium within the landscapes they occupy, although predicting their potential distributions is often of critical importance to natural resource managers. We determined if the accuracy of species distribution models differed between 17 native and 17 introduced riparian plant species in the western United States. We also assessed if model accuracy was associated with both environmental and biological factors that can influence dispersal. We used Random Forests to model species distributions and linear regression to determine if model accuracy was associated with dispersal‐related traits. Model accuracy for introduced species was higher than that for native species. Dispersal‐related traits did not affect model accuracy or improvement, though two other traits, family affiliation and rarity on the landscape, did have an effect. Distance‐based measures of dispersal potential improved model fit equally for both native and introduced species and for species with a variety of dispersal traits, suggesting that the importance of regional propagule pressure is relatively constant across species with different dispersal opportunities. Several lines of future questioning are suggested by our results, including why introduced species may in some cases produce more accurate distribution models than native species and how species dispersal traits relate to distribution model accuracy at different spatial scales.     

An interpretation for the ROC curve and inference using GLM procedures

The accuracy of a medical diagnostic test is often summarized in a receiver operating characteristic (ROC) curve. This paper puts forth an interpretation for each point on the ROC curve as being a conditional probability of a test result from a random diseased subject exceeding that from a random nondiseased subject. This interpretation gives rise to new methods for making inference about ROC curves. It is shown that inference can be achieved with binary regression techniques applied to indicator variables constructed from pairs of test results, one component of the pair being from a diseased subject and the other from a nondiseased subject. Within the generalized linear model (GLM) binary regression framework, ROC curves can be estimated, and we highlight a new semiparametric estimator. Covariate effects can also be evaluated with the GLM models. The methodology is applied to a pancreatic cancer dataset where we use the regression framework to compare two different serum biomarkers. Asymptotic distribution theory is developed to facilitate inference and to provide insight into factors influencing variability of estimated model parameters.     

Least squares regression methods for clustered ROC data with discrete covariates

The receiver operating characteristic (ROC) curve is a popular tool to evaluate and compare the accuracy of diagnostic tests to distinguish the diseased group from the nondiseased group when test results from tests are continuous or ordinal. A complicated data setting occurs when multiple tests are measured on abnormal and normal locations from the same subject and the measurements are clustered within the subject. Although least squares regression methods can be used for the estimation of ROC curve from correlated data, how to develop the least squares methods to estimate the ROC curve from the clustered data has not been studied. Also, the statistical properties of the least squares methods under the clustering setting are unknown. In this article, we develop the least squares ROC methods to allow the baseline and link functions to differ, and more importantly, to accommodate clustered data with discrete covariates. The methods can generate smooth ROC curves that satisfy the inherent continuous property of the true underlying curve. The least squares methods are shown to be more efficient than the existing nonparametric ROC methods under appropriate model assumptions in simulation studies. We apply the methods to a real example in the detection of glaucomatous deterioration. We also derive the asymptotic properties of the proposed methods.     

Measuring and comparing the accuracy of species distribution models with presence–absence data

Species distribution models have been widely used to predict species distributions for various purposes, including conservation planning, and climate change impact assessment. The success of these applications relies heavily on the accuracy of the models. Various measures have been proposed to assess the accuracy of the models. Rigorous statistical analysis should be incorporated in model accuracy assessment. However, since relevant information about the statistical properties of accuracy measures is scattered across various disciplines, ecologists find it difficult to select the most appropriate ones for their research. In this paper, we review accuracy measures that are currently used in species distribution modelling (SDM), and introduce additional metrics that have potential applications in SDM. For the commonly used measures (which are also intensively studied by statisticians), including overall accuracy, sensitivity, specificity, kappa, and area and partial area under the ROC curves, promising methods to construct confidence intervals and statistically compare the accuracy between two models are given. For other accuracy measures, methods to estimate standard errors are given, which can be used to construct approximate confidence intervals. We also suggest that as general tools, computer‐intensive methods, especially bootstrap and randomization methods can be used in constructing confidence intervals and statistical tests if suitable analytic methods cannot be found. Usually, these computer‐intensive methods provide robust results.     

Combining geostatistical models and remotely sensed data to improve tropical tree richness mapping

Information on the spatial distribution and composition of biological communities is essential in designing effective strategies for biodiversity conservation and management. Reliable maps of species richness across the landscape can be useful tools for these purposes. Acquiring such information through traditional survey techniques is costly and logistically difficult. The kriging interpolation method has been widely used as an alternative to predict spatial distributions of species richness, as long as the data are spatially dependent. However, even when this requirement is met, researchers often have few sampled sites in relation to the area to be mapped. Remote sensing provides an inexpensive means to derive complete spatial coverage for large areas and can be extremely useful for estimating biodiversity. The aim of this study was to combine remotely sensed data with kriging estimates (hybrid procedures) to evaluate the possibility of improving the accuracy of tree species richness maps. We did this through the comparison of the predictive performance of three hybrid geostatistical procedures, based on tree species density recorded in 141 sampling quadrats: co-kriging (COK), kriging with external drift (KED), and regression kriging (RK). Reflectance values of spectral bands, computed NDVI and texture measurements of Landsat 7 TM imagery were used as ancillary variables in all methods. The R² values of the models increased from 0.35 for ordinary kriging to 0.41 for COK, and from 0.39 for simple regression estimates to 0.52 and 0.53 when using simple KED and RK, respectively. The R² values of the models also increased from 0.60 for multiple regression estimates to 0.62 and 0.66 when using multiple KED and RK, respectively. Overall, our results demonstrate that these procedures are capable of greatly improving estimation accuracy, with multivariate RK being clearly superior, because it produces the most accurate predictions, and because of its flexibility in modeling multivariate relationships between tree richness and remotely sensed data. We conclude that this is a valuable tool for guiding future efforts aimed at conservation and management of highly diverse tropical forests.     

Uncertainty of bioclimate envelope models based on the geographical distribution of species

Aim We explored the effects of prevalence, latitudinal range and spatial autocorrelation of species distribution patterns on the accuracy of bioclimate envelope models of butterflies. Location Finland, northern Europe. Methods The data of a national butterfly atlas survey (NAFI) carried out in 1991–2003 with a resolution of 10 × 10 km were used in the analyses. Generalized additive models (GAM) were constructed, for each of 98 species, to estimate the probability of occurrence as a function of climate variables. Model performance was measured using the area under the curve (AUC) of a receiver operating characteristic (ROC) plot. Observed differences in modelling accuracy among species were related to the species’ geographical attributes using multivariate GAM. Results Accuracies of the climate–butterfly models varied from low to very high (AUC values 0.59–0.99), with a mean of 0.79. The modelling performance was related negatively to the latitudinal range and prevalence, and positively to the spatial autocorrelation of the species distribution. These three factors accounted for 75.2% of the variation in the modelling accuracy. Species at the margin of their range or with low prevalence were better predicted than widespread species, and species with clumped distributions better than scattered dispersed species. Main conclusions The results from this study indicate that species’ geographical attributes highly influence the behaviour and uncertainty of species–climate models, which should be taken into account in biogeographical modelling studies and assessments of climate change impacts.     

The analysis of placement values for evaluating discriminatory measures

The idea of using measurements such as biomarkers, clinical data, or molecular biology assays for classification and prediction is popular in modern medicine. The scientific evaluation of such measures includes assessing the accuracy with which they predict the outcome of interest. Receiver operating characteristic curves are commonly used for evaluating the accuracy of diagnostic tests. They can be applied more broadly, indeed to any problem involving classification to two states or populations (D= 0 or 1). We show that the ROC curve can be interpreted as a cumulative distribution function for the discriminatory measure Y in the affected population (D= 1) after Y has been standardized to the distribution in the reference population (D= 0). The standardized values are called placement values. If the placement values have a uniform(0, 1) distribution, then Y is not discriminatory, because its distribution in the affected population is the same as that in the reference population. The degree to which the distribution of the standardized measure differs from uniform(0, 1) is a natural way to characterize the discriminatory capacity of Y and provides a nontraditional interpretation for the ROC curve. Statistical methods for making inference about distribution functions therefore motivate new approaches to making inference about ROC curves. We demonstrate this by considering the ROC-GLM regression model and observing that it is equivalent to a regression model for the distribution of placement values. The likelihood of the placement values provides a new approach to ROC parameter estimation that appears to be more efficient than previously proposed methods. The method is applied to evaluate a pulmonary function measure in cystic fibrosis patients as a predictor of future occurrence of severe acute pulmonary infection requiring hospitalization. Finally, we note the relationship between regression models for the mean placement value and recently proposed models for the area under the ROC curve which is the classic summary index of discrimination.     

An evaluation of condition indices and predictive models for noninvasive estimates of lipid mass of migrating Common Yellowthroats, Ovenbirds, and Swainson's Thrushes

ABSTRACT.   Noninvasive methods of measuring lipid mass in birds are widely used, but not frequently evaluated. I evaluated the ability of three noninvasive indicators of fat content (fat scores, body mass, body mass/wing chord) and regression models to predict lipid mass in two migratory songbirds previously unexamined in this context—Common Yellowthroat ( Geothlypis trichas ) and Ovenbird ( Seiurus aurocapillus ). I also examined the accuracy of these methods for Swainson's Thrushes ( Catharus ustulatus ) for comparison to a previous study. Fat score, body mass, and body mass/wing chord were highly correlated with chemically extracted lipid mass in each species. In all three species, birds with no visible subcutaneous fat possessed considerable quantities of fat, ranging from 9.8 to 19.7% of total dry body mass. Forward-selected regression models explained 69−87% of lipid mass variation, with prediction errors of 14.6−27.5%. An existing predictive model for the Swainson's Thrush overestimated lipid mass by an average of 92%. Fat score, body mass, and the regression models generated here are reliable predictors of lipid mass in two of the three migrating species examined. The accuracy of the methods, in addition to their low cost and simplicity, justifies their continued use in field studies of birds.