Insights into the area under the receiver operating characteristic curve (AUC) as a discrimination measure in species distribution modelling

Aim The area under the receiver operating characteristic (ROC) curve (AUC) is a widely used statistic for assessing the discriminatory capacity of species distribution models. Here, I used simulated data to examine the interdependence of the AUC and classical discrimination measures (sensitivity and specificity) derived for the application of a threshold. I shall further exemplify with simulated data the implications of using the AUC to evaluate potential versus realized distribution models. Innovation After applying the threshold that makes sensitivity and specificity equal, a strong relationship between the AUC and these two measures was found. This result is corroborated with real data. On the other hand, the AUC penalizes the models that estimate potential distributions (the regions where the species could survive and reproduce due to the existence of suitable environmental conditions), and favours those that estimate realized distributions (the regions where the species actually lives). Main conclusions Firstly, the independence of the AUC from the threshold selection may be irrelevant in practice. This result also emphasizes the fact that the AUC assumes nothing about the relative costs of errors of omission and commission. However, in most real situations this premise may not be optimal. Measures derived from a contingency table for different cost ratio scenarios, together with the ROC curve, may be more informative than reporting just a single AUC value. Secondly, the AUC is only truly informative when there are true instances of absence available and the objective is the estimation of the realized distribution. When the potential distribution is the goal of the research, the AUC is not an appropriate performance measure because the weight of commission errors is much lower than that of omission errors.     

Semi-parametric area under the curve regression method for diagnostic studies with ordinal data

The classification accuracy of new diagnostic tests is based on receiver operating characteristic (ROC) curves. The area under the ROC curve (AUC) is one of the well-accepted summary measures for describing the accuracy of diagnostic tests. The AUC summary measure can vary by patient and testing characteristics. Thus, the performance of the test may be different in certain subpopulation of patients and readers. For this purpose, we propose a direct semi-parametric regression model for the non-parametric AUC measure for ordinal data while accounting for discrete and continuous covariates. The proposed method can be used to estimate the AUC value under degenerate data where certain rating categories are not observed. We will discuss the non-standard asymptotic theory, since the estimating functions were based on cross-correlated random variables. Simulation studies based on different classification models showed that the proposed model worked reasonably well with small percent bias and percent mean-squared error. The proposed method was applied to the prostate cancer study to estimate the AUC for four readers, and the carotid vessel study with age, gender, history of previous stroke, and total number of risk factors as covariates, to estimate the accuracy of the diagnostic test in the presence of subject-level covariates.     

ROC曲线分析在评价入侵物种分布模型中的应用

生态位模型（ecological niche models,ENMs）已广泛应用于物种潜在分布区预测,ENMs的应用也为外来入侵物种的风险分析提供了重要的定量化分析工具,但如何评价不同模型之间的预测效果成了当今研究的热点问题。本文介绍了受试者工作特征（ROC）曲线分析在评价不同生态位模型预测效果中的应用原理和分析方法,并以一种植物病原线虫-相似穿孔线虫（Radopholus similis）为例,应用ROC曲线分析法对其5种模型（BIOCLIM,CLIMEX,DOMAIN,GARP,MAXENT）的预测结果进行了比较分析。5种模型的ROC曲线下面积AUC（Area Under Curve）值分别为0.810,0.758,0.921,0.903和0.950,以MAXENT模型的AUC值最大,表明其预测效果最好;方差分析结果表明,除GARP与DOMAIN模型之间AUC值差异不显著外,其余各模型之间差异显著。     

Effect of species rarity on the accuracy of species distribution models for reptiles and amphibians in southern California

Aim Several studies have found that more accurate predictive models of species’ occurrences can be developed for rarer species; however, one recent study found the relationship between range size and model performance to be an artefact of sample prevalence, that is, the proportion of presence versus absence observations in the data used to train the model. We examined the effect of model type, species rarity class, species’ survey frequency, detectability and manipulated sample prevalence on the accuracy of distribution models developed for 30 reptile and amphibian species. Location Coastal southern California, USA. Methods Classification trees, generalized additive models and generalized linear models were developed using species presence and absence data from 420 locations. Model performance was measured using sensitivity, specificity and the area under the curve (AUC) of the receiver‐operating characteristic (ROC) plot based on twofold cross‐validation, or on bootstrapping. Predictors included climate, terrain, soil and vegetation variables. Species were assigned to rarity classes by experts. The data were sampled to generate subsets with varying ratios of presences and absences to test for the effect of sample prevalence. Join count statistics were used to characterize spatial dependence in the prediction errors. Results Species in classes with higher rarity were more accurately predicted than common species, and this effect was independent of sample prevalence. Although positive spatial autocorrelation remained in the prediction errors, it was weaker than was observed in the species occurrence data. The differences in accuracy among model types were slight. Main conclusions Using a variety of modelling methods, more accurate species distribution models were developed for rarer than for more common species. This was presumably because it is difficult to discriminate suitable from unsuitable habitat for habitat generalists, and not as an artefact of the effect of sample prevalence on model estimation.     

Does accounting for imperfect detection improve species distribution models?

Models of species distributions are increasingly being used to address a variety of problems in conservation biology. In many applications, perfect or constant detectability of species, given presence, is assumed. While this problem has been acknowledged and addressed through the development of occupancy models, we still know little regarding whether addressing the potential for imperfect detection improves the predictive performance of species distribution models in nature. Here, we contrast logistic regression models of species occurrence that do not correct for detectability to hierarchical occupancy models that explicitly estimate and adjust for detectability, and maximum entropy models that attempt to circumvent the detectability problem by using data from known presence locations only. We use a large‐scale, long‐term monitoring database across western Montana and northern Idaho to contrast these models for nine landbird species that cover a broad spectrum in detectability. Overall, occupancy models were similar to or better than other approaches in terms of predictive accuracy, as measured by the Area Under the ROC Curve (AUC) and Kappa, with maximum entropy tending to provide the lowest predictive accuracy. Models varied in the types of errors associated with predictions, such that some model approaches may be preferred over others in certain situations. As expected, predictive performance varied across a gradient in species detectability, with logistic regression providing lower relative performance for less detectable species and Maxent providing lower performance for highly detectable species. We conclude by discussing the advantages and limitations to each approach for developing large‐scale species distribution models.     

Assessing the effect of prevalence on the predictive performance of species distribution models using simulated data

Aim The proportion of sampled sites where a species is present is known as prevalence. Empirical studies have shown that prevalence can affect the predictive performance of species distribution models. This paper uses simulated species data to examine how prevalence and the form of species environmental dependence affect the assessment of the predictive performance of models. Methods Simulated species data were based on various functions of simulated environmental data with differing degrees of spatial correlation. Seven model performance measures – sensitivity, specificity, class‐average (CA), overall prediction success, kappa (κ), normalized mutual information (NMI) and area under the receiver operating characteristic curve (AUC) – were applied to species models fitted by three regression methods. The response of the performance measures to prevalence was then assessed. Three probability threshold selection methods used to convert fitted logistic model values to presence or absence were also assessed. Results The study shows that the extent to which prevalence affects model performance depends on the modelling technique and its degree of success in capturing dominant environmental determinants. It also depends on the statistic used to measure model performance and the probability threshold method. The response based on κ generally preferred models with medium prevalence. All performance measures were least affected by prevalence when the probability threshold was chosen to maximize predictive performance or was based directly on prevalence. In these cases, the responses based on AUC, CA and NMI generally preferred models with small or large prevalence. Main conclusions The effect of prevalence on the predictive performance of species distribution models has a methodological basis. Relevant factors include the success of the fitted distribution model in capturing the dominant environmental determinant, the model performance measure and the probability threshold selection method. The fixed probability threshold method yields a marked response of model performance to prevalence and is therefore not recommended. The study explains previous empirical results obtained with real data.     

Uncertainty of bioclimate envelope models based on the geographical distribution of species

Aim We explored the effects of prevalence, latitudinal range and spatial autocorrelation of species distribution patterns on the accuracy of bioclimate envelope models of butterflies. Location Finland, northern Europe. Methods The data of a national butterfly atlas survey (NAFI) carried out in 1991–2003 with a resolution of 10 × 10 km were used in the analyses. Generalized additive models (GAM) were constructed, for each of 98 species, to estimate the probability of occurrence as a function of climate variables. Model performance was measured using the area under the curve (AUC) of a receiver operating characteristic (ROC) plot. Observed differences in modelling accuracy among species were related to the species’ geographical attributes using multivariate GAM. Results Accuracies of the climate–butterfly models varied from low to very high (AUC values 0.59–0.99), with a mean of 0.79. The modelling performance was related negatively to the latitudinal range and prevalence, and positively to the spatial autocorrelation of the species distribution. These three factors accounted for 75.2% of the variation in the modelling accuracy. Species at the margin of their range or with low prevalence were better predicted than widespread species, and species with clumped distributions better than scattered dispersed species. Main conclusions The results from this study indicate that species’ geographical attributes highly influence the behaviour and uncertainty of species–climate models, which should be taken into account in biogeographical modelling studies and assessments of climate change impacts.     

Spind: a package for computing spatially corrected accuracy measures

Using an appropriate accuracy measure is essential for assessing prediction accuracy in species distribution modelling. Therefore, model evaluation as an analytical uncertainty is a challenging problem. Although a variety of accuracy measures for the assessment of prediction errors in presence/absence models is available, there is a lack of spatial accuracy measures, i.e. measures that are sensitive to the spatial arrangement of the predictions. We present ‘spind’, a new software package (based on the R software program) that provides spatial performance measures for grid‐based models. These accuracy measures are generalized, spatially corrected versions of the classical ones, thus enabling comparisons between them. Our method for evaluation consists of the following steps: 1) incorporate additional autocorrelation until spatial autocorrelation in predictions and actuals is balanced, 2) cross‐classify predictions and adjusted actuals in a 4 × 4 contingency table, 3) use a refined weighting pattern for errors, and 4) calculate weighted Kappa, sensitivity, specificity and subsequently ROC, AUC, TSS to get spatially corrected indices. To illustrate the impact of our spatial method we present an example of simulated data as well as an example of presence/absence data of the plant species Dianthus carthusianorum across Germany. Our analysis includes a statistic for the comparison of spatial and classical (non‐spatial) indices. We find that our spatial indices tend to result in higher values than classical ones. These differences are statistically significant at medium and high autocorrelation levels. We conclude that these spatial accuracy measures may contribute to evaluate prediction errors in presence/absence models, especially in case of medium or high degree of similarity of adjacent data, i.e. aggregated (clumped) or continuous species distributions.     

Threshold-dependence as a desirable attribute for discrimination assessment: implications for the evaluation of species distribution models

Species distribution modelling has become a common approach in ecology in the last decades. As in any modelling exercise, evaluation of the predicted suitability surfaces is a key process, and the area under the receiver operating characteristic (ROC) curve (AUC) has become the most popular statistic for this purpose. A close covariation between the AUC and threshold-dependent discrimination measures (sensitivity Se and specificity Sp) raises into question the advantage of the threshold-independence of the AUC. In this study, the relationship between the AUC and several threshold-dependent discrimination measures is characterized in detail, and the sensitivity of the pattern to variations in the shape of the ROC curve is assessed. Hypothetical suitability values, coming from normal and skew-normal distributions, were simulated for both instances of presence and absence. The flexibility of the skew-normal distribution allowed for the simulation of a wide range of ROC curve configurations. The relationship between the AUC and threshold-dependent measures was graphically assessed; independently of the ROC curve shape, a nonlinear asymptotic relationship between the AUC and Se (and Sp) was obtained after applying the threshold that makes Se = Sp. A nonlinear asymptotic relationship between the AUC and the Youden index was also reported. These results imply that the AUC does not appropriately measure changes in the discrimination of models, and it is especially incapable of distinguishing between models with high discrimination capacity. Se or Sp derived from the application of the threshold that makes them equal is a preferred measure of discrimination power. Together with the rate of false positives and negatives, and with the prevalence of the species, these statistics provide more information about the discrimination capacity of the models than the AUC.     

Assessing models for genetic prediction of complex traits: a comparison of visualization and quantitative methods

Background

In silico models have recently been created in order to predict which genetic variants are more likely to contribute to the risk of a complex trait given their functional characteristics. However, there has been no comprehensive review as to which type of predictive accuracy measures and data visualization techniques are most useful for assessing these models.

Methods

We assessed the performance of the models for predicting risk using various methodologies, some of which include: receiver operating characteristic (ROC) curves, histograms of classification probability, and the novel use of the quantile-quantile plot. These measures have variable interpretability depending on factors such as whether the dataset is balanced in terms of numbers of genetic variants classified as risk variants versus those that are not.

Results

We conclude that the area under the curve (AUC) is a suitable starting place, and for models with similar AUCs, violin plots are particularly useful for examining the distribution of the risk scores.

Electronic supplementary material

The online version of this article (doi:10.1186/s12864-015-1616-z) contains supplementary material, which is available to authorized users.     

The analysis of placement values for evaluating discriminatory measures

The idea of using measurements such as biomarkers, clinical data, or molecular biology assays for classification and prediction is popular in modern medicine. The scientific evaluation of such measures includes assessing the accuracy with which they predict the outcome of interest. Receiver operating characteristic curves are commonly used for evaluating the accuracy of diagnostic tests. They can be applied more broadly, indeed to any problem involving classification to two states or populations (D= 0 or 1). We show that the ROC curve can be interpreted as a cumulative distribution function for the discriminatory measure Y in the affected population (D= 1) after Y has been standardized to the distribution in the reference population (D= 0). The standardized values are called placement values. If the placement values have a uniform(0, 1) distribution, then Y is not discriminatory, because its distribution in the affected population is the same as that in the reference population. The degree to which the distribution of the standardized measure differs from uniform(0, 1) is a natural way to characterize the discriminatory capacity of Y and provides a nontraditional interpretation for the ROC curve. Statistical methods for making inference about distribution functions therefore motivate new approaches to making inference about ROC curves. We demonstrate this by considering the ROC-GLM regression model and observing that it is equivalent to a regression model for the distribution of placement values. The likelihood of the placement values provides a new approach to ROC parameter estimation that appears to be more efficient than previously proposed methods. The method is applied to evaluate a pulmonary function measure in cystic fibrosis patients as a predictor of future occurrence of severe acute pulmonary infection requiring hospitalization. Finally, we note the relationship between regression models for the mean placement value and recently proposed models for the area under the ROC curve which is the classic summary index of discrimination.     

Exploration of Analysis Methods for Diagnostic Imaging Tests: Problems with ROC AUC and Confidence Scores in CT Colonography

Background

Different methods of evaluating diagnostic performance when comparing diagnostic tests may lead to different results. We compared two such approaches, sensitivity and specificity with area under the Receiver Operating Characteristic Curve (ROC AUC) for the evaluation of CT colonography for the detection of polyps, either with or without computer assisted detection.

Methods

In a multireader multicase study of 10 readers and 107 cases we compared sensitivity and specificity, using radiological reporting of the presence or absence of polyps, to ROC AUC calculated from confidence scores concerning the presence of polyps. Both methods were assessed against a reference standard. Here we focus on five readers, selected to illustrate issues in design and analysis. We compared diagnostic measures within readers, showing that differences in results are due to statistical methods.

Results

Reader performance varied widely depending on whether sensitivity and specificity or ROC AUC was used. There were problems using confidence scores; in assigning scores to all cases; in use of zero scores when no polyps were identified; the bimodal non-normal distribution of scores; fitting ROC curves due to extrapolation beyond the study data; and the undue influence of a few false positive results. Variation due to use of different ROC methods exceeded differences between test results for ROC AUC.

Conclusions

The confidence scores recorded in our study violated many assumptions of ROC AUC methods, rendering these methods inappropriate. The problems we identified will apply to other detection studies using confidence scores. We found sensitivity and specificity were a more reliable and clinically appropriate method to compare diagnostic tests.     

Habitat Suitability Models,for ecological study of the alpine marmot in the central Italian Alps

Habitat Suitability Models (HSMs) are central tools in physical and environmental planning because they are able to predict potential distribution of species. In this study two spatial methods were compared to define the HSM for Marmota marmota L. in the Adamello Brenta Nature Park in the Italian central southern part of the Alps. The first model was based on Weighted Linear Combination (WLC), a nonparametric overlay procedure, the second one was a parametric method based on logistic regression algorithm. Data collected in two sample areas were used to build the models, while data recorded along linear transects evenly distributed throughout the study area were used for testing the model performance. The models were validated calculating the Area Under the Curve (AUC) of the Relative Operating Characteristics (ROC). The two models had similar performances, but the accuracy of the WLC model was slightly higher than the logistic model (AUC of 0.833 instead of 0.821). The habitat preferences of Marmota marmota L. were also investigated with Jacobs index and chi-square test and compared with other studies in different environments. The results demonstrated that the relative importance of environmental factors for Marmota marmota L. changes depending on local conditions.     

ROC曲线形状在生态位模型评价中的重要性——以美国白蛾为例

【目的】生态位模型在生物地理学、入侵生物学和保护生物学中具有广泛的应用,被越来越多地用于预测物种潜在分布和现实分布的研究中。本文以美国白蛾为例介绍pROC方案在生态位模型评价中的应用及其注意事项,以期对物种潜在分布预测进行合理的评价,促进生态位模型在我国的合理运用和发展。【方法】介绍ROC曲线和AUC值基本原理,总结其在生态位模型评价中的应用,从物种存在分布点和不存在分布点的可信度出发,分析AUC值用于模型评价的优点和不足,最后介绍局部受试者工作特征曲线的线下面积方案(pROC方案)来弥补传统AUC值的不足。【结果】AUC值虽独立于阈值,但因其综合灵敏度和特异度,而屏蔽这2个指标各自的特征,不能分别评估预测结果的灵敏度和特异度,同时对遗漏率和记账错率不能进行权衡,会误导使用者对模型的评价。与AUC值相比,ROC曲线的形状更具有价值,蕴含丰富的模型评价信息。【结论】模型评价需要将灵敏度和特异度区别对待,ROC曲线形状比AUC值在生态位模型评价中更为重要,pROC方案相对于传统AUC值具有优势,但容易对过度模拟做出不当判断。模型评价与作者研究目的密切相关:当以预测物种潜在分布为目的时(如入侵物种潜在分布、气候变化对物种分布的影响和谱系生物地理学),模型评价应当给予灵敏度(或者遗漏率)更多的权重;当以预测物种现实分布为目的时(如保护区界定和濒危物种引入),模型评价应当给予灵敏度和特异度同等的权重。     

A comparison of absolute performance of different correlative and mechanistic species distribution models in an independent area

To investigate the comparative abilities of six different bioclimatic models in an independent area, utilizing the distribution of eight different species available at a global scale and in Australia. Global scale and Australia. We tested a variety of bioclimatic models for eight different plant species employing five discriminatory correlative species distribution models (SDMs) including Generalized Linear Model (GLM), MaxEnt, Random Forest (RF), Boosted Regression Tree (BRT), Bioclim, together with CLIMEX (CL) as a mechanistic niche model. These models were fitted using a training dataset of available global data, but with the exclusion of Australian locations. The capabilities of these techniques in projecting suitable climate, based on independent records for these species in Australia, were compared. Thus, Australia is not used to calibrate the models and therefore it is as an independent area regarding geographic locations. To assess and compare performance, we utilized the area under the receiver operating characteristic (ROC) curves (AUC), true skill statistic (TSS), and fractional predicted areas for all SDMs. In addition, we assessed satisfactory agreements between the outputs of the six different bioclimatic models, for all eight species in Australia. The modeling method impacted on potential distribution predictions under current climate. However, the utilization of sensitivity and the fractional predicted areas showed that GLM, MaxEnt, Bioclim, and CL had the highest sensitivity for Australian climate conditions. Bioclim calculated the highest fractional predicted area of an independent area, while RF and BRT were poor. For many applications, it is difficult to decide which bioclimatic model to use. This research shows that variable results are obtained using different SDMs in an independent area. This research also shows that the SDMs produce different results for different species; for example, Bioclim may not be good for one species but works better for other species. Also, when projecting a “large” number of species into novel environments or in an independent area, the selection of the “best” model/technique is often less reliable than an ensemble modeling approach. In addition, it is vital to understand the accuracy of SDMs' predictions. Further, while TSS, together with fractional predicted areas, are appropriate tools for the measurement of accuracy between model results, particularly when undertaking projections on an independent area, AUC has been proved not to be. Our study highlights that each one of these models (CL, Bioclim, GLM, MaxEnt, BRT, and RF) provides slightly different results on projections and that it may be safer to use an ensemble of models.     

Measures and limits of models of fixation selection

Models of fixation selection are a central tool in the quest to understand how the human mind selects relevant information. Using this tool in the evaluation of competing claims often requires comparing different models' relative performance in predicting eye movements. However, studies use a wide variety of performance measures with markedly different properties, which makes a comparison difficult. We make three main contributions to this line of research: First we argue for a set of desirable properties, review commonly used measures, and conclude that no single measure unites all desirable properties. However the area under the ROC curve (a classification measure) and the KL-divergence (a distance measure of probability distributions) combine many desirable properties and allow a meaningful comparison of critical model performance. We give an analytical proof of the linearity of the ROC measure with respect to averaging over subjects and demonstrate an appropriate correction of entropy-based measures like KL-divergence for small sample sizes in the context of eye-tracking data. Second, we provide a lower bound and an upper bound of these measures, based on image-independent properties of fixation data and between subject consistency respectively. Based on these bounds it is possible to give a reference frame to judge the predictive power of a model of fixation selection. We provide open-source python code to compute the reference frame. Third, we show that the upper, between subject consistency bound holds only for models that predict averages of subject populations. Departing from this we show that incorporating subject-specific viewing behavior can generate predictions which surpass that upper bound. Taken together, these findings lay out the required information that allow a well-founded judgment of the quality of any model of fixation selection and should therefore be reported when a new model is introduced.     

预测物种潜在分布区——比较SVM与GARP

 物种分布与环境因子之间存在着紧密的联系,因此利用环境因子作为预测物种分布模型的变量是当前最普遍的建模思路,但是绝大多数物种分 布预测模型都遇到了难以解决的“高维小样本"问题。该研究通过理论和实践证明,基于结构风险最小化原理的支持向量机(Support vector machine, SVM)算法非常适合“高维小样本"的分类问题。以20种杜鹃花属(Rhododendron)中国特有种为检验对象,利用标本数据和11个1 km×1 km的栅格环境数据层作为模型变量,预测其在中国的潜在分布区,并通过全面的模型评估——专家评估,受试者工作特征(Receiver operator characteristic, ROC)曲线和曲线下方面积(Area under the curve, AUC)——来比较模型的性能。我们实现了以SVM为核心的物种分布预测 系统,并且通过试验证明其无论在计算速度还是预测效果上都远远优于当前广泛使用的规则集合预测的遗传算法(Algorithm for rule-set prediction, GARP)预测系统。     

我国林火发生预测模型研究进展

通过文献回顾,总结了国内林火发生预测模型的研究现状,并从林火发生驱动因子、林火发生概率预测模型、林火发生频次预测模型和模型检验方法等方面进行归纳分析。得出以下结论: 1)气象、地形、植被、可燃物、人类活动等因素是影响林火发生及模型预测精度的主要驱动因子;2)林火发生概率模型中,地理加权逻辑斯蒂回归模型考虑了变量之间的空间相关性,Gompit回归模型适宜非对称结构的林火数据,随机森林模型不需要多重共线性检验,在避免过度拟合的同时提高了预测精度,是林火发生概率预测模型的优选方法之一;3)林火发生频次模型中,负二项回归模型更适合对过度离散数据进行模拟,零膨胀模型和栅栏模型可以处理林火数据中包含大量零值的问题;4)ROC检验、AIC检验、似然比检验和Wald检验方法是林火概率和频次模型的常用检验方法。林火发生预测模型研究仍是我国当前林火管理工作的重点,预测模型的选择需要依据不同地区林火数据特点。此外,构建林火预测模型时需要考虑更多的影响因素,以提高模型预测精度;未来,需要进一步探索其他数学模型在林火发生预测中的应用,不断提高林火发生预测模型的准确度。     

Robust Estimation of Area Under ROC Curve Using Auxiliary Variables in the Presence of Missing Biomarker Values

Summary In medical research, the receiver operating characteristic (ROC) curves can be used to evaluate the performance of biomarkers for diagnosing diseases or predicting the risk of developing a disease in the future. The area under the ROC curve (ROC AUC), as a summary measure of ROC curves, is widely utilized, especially when comparing multiple ROC curves. In observational studies, the estimation of the AUC is often complicated by the presence of missing biomarker values, which means that the existing estimators of the AUC are potentially biased. In this article, we develop robust statistical methods for estimating the ROC AUC and the proposed methods use information from auxiliary variables that are potentially predictive of the missingness of the biomarkers or the missing biomarker values. We are particularly interested in auxiliary variables that are predictive of the missing biomarker values. In the case of missing at random (MAR), that is, missingness of biomarker values only depends on the observed data, our estimators have the attractive feature of being consistent if one correctly specifies, conditional on auxiliary variables and disease status, either the model for the probabilities of being missing or the model for the biomarker values. In the case of missing not at random (MNAR), that is, missingness may depend on the unobserved biomarker values, we propose a sensitivity analysis to assess the impact of MNAR on the estimation of the ROC AUC. The asymptotic properties of the proposed estimators are studied and their finite‐sample behaviors are evaluated in simulation studies. The methods are further illustrated using data from a study of maternal depression during pregnancy.