Combining multiple markers for classification using ROC

In biomedical studies, it is of great interest to develop methodologies for combining multiple markers for the purpose of disease classification. The receiving operating characteristic (ROC) technique has been widely used, where classification performance can be measured with the area under the ROC curve (AUC). In this article, we study a ROC-based method for effectively combining multiple markers for disease classification. We propose a sigmoid AUC (SAUC) estimator that maximizes the sigmoid approximation of the empirical AUC. The SAUC estimator is computationally affordable, n(1/2)-consistent and achieves the same asymptotic efficiency as the AUC estimator. Inference based on the weighted bootstrap is investigated. We also propose Monte Carlo methods to assess the overall prediction performance and the relative importance of individual markers. Finite sample performance is evaluated using simulation studies and two public data sets.     

Robust Estimation of Area Under ROC Curve Using Auxiliary Variables in the Presence of Missing Biomarker Values

Summary In medical research, the receiver operating characteristic (ROC) curves can be used to evaluate the performance of biomarkers for diagnosing diseases or predicting the risk of developing a disease in the future. The area under the ROC curve (ROC AUC), as a summary measure of ROC curves, is widely utilized, especially when comparing multiple ROC curves. In observational studies, the estimation of the AUC is often complicated by the presence of missing biomarker values, which means that the existing estimators of the AUC are potentially biased. In this article, we develop robust statistical methods for estimating the ROC AUC and the proposed methods use information from auxiliary variables that are potentially predictive of the missingness of the biomarkers or the missing biomarker values. We are particularly interested in auxiliary variables that are predictive of the missing biomarker values. In the case of missing at random (MAR), that is, missingness of biomarker values only depends on the observed data, our estimators have the attractive feature of being consistent if one correctly specifies, conditional on auxiliary variables and disease status, either the model for the probabilities of being missing or the model for the biomarker values. In the case of missing not at random (MNAR), that is, missingness may depend on the unobserved biomarker values, we propose a sensitivity analysis to assess the impact of MNAR on the estimation of the ROC AUC. The asymptotic properties of the proposed estimators are studied and their finite‐sample behaviors are evaluated in simulation studies. The methods are further illustrated using data from a study of maternal depression during pregnancy.     

Exploration of Analysis Methods for Diagnostic Imaging Tests: Problems with ROC AUC and Confidence Scores in CT Colonography

Background

Different methods of evaluating diagnostic performance when comparing diagnostic tests may lead to different results. We compared two such approaches, sensitivity and specificity with area under the Receiver Operating Characteristic Curve (ROC AUC) for the evaluation of CT colonography for the detection of polyps, either with or without computer assisted detection.

Methods

In a multireader multicase study of 10 readers and 107 cases we compared sensitivity and specificity, using radiological reporting of the presence or absence of polyps, to ROC AUC calculated from confidence scores concerning the presence of polyps. Both methods were assessed against a reference standard. Here we focus on five readers, selected to illustrate issues in design and analysis. We compared diagnostic measures within readers, showing that differences in results are due to statistical methods.

Results

Reader performance varied widely depending on whether sensitivity and specificity or ROC AUC was used. There were problems using confidence scores; in assigning scores to all cases; in use of zero scores when no polyps were identified; the bimodal non-normal distribution of scores; fitting ROC curves due to extrapolation beyond the study data; and the undue influence of a few false positive results. Variation due to use of different ROC methods exceeded differences between test results for ROC AUC.

Conclusions

The confidence scores recorded in our study violated many assumptions of ROC AUC methods, rendering these methods inappropriate. The problems we identified will apply to other detection studies using confidence scores. We found sensitivity and specificity were a more reliable and clinically appropriate method to compare diagnostic tests.     

Empirical likelihood inference for the area under the ROC curve

For a continuous-scale diagnostic test, the most commonly used summary index of the receiver operating characteristic curve (ROC) is the area under the curve (AUC) that measures the accuracy of the diagnostic test. In this article, we propose an empirical likelihood (EL) approach for the inference on the AUC. First we define an EL ratio for the AUC and show that its limiting distribution is a scaled chi-square distribution. We then obtain an EL-based confidence interval for the AUC using the scaled chi-square distribution. This EL inference for the AUC can be extended to stratified samples, and the resulting limiting distribution is a weighted sum of independent chi-square distributions. Additionally we conduct simulation studies to compare the relative performance of the proposed EL-based interval with the existing normal approximation-based intervals and bootstrap intervals for the AUC.     

A boosting method for maximizing the partial area under the ROC curve

Background  

The receiver operating characteristic (ROC) curve is a fundamental tool to assess the discriminant performance for not only a single marker but also a score function combining multiple markers. The area under the ROC curve (AUC) for a score function measures the intrinsic ability for the score function to discriminate between the controls and cases. Recently, the partial AUC (pAUC) has been paid more attention than the AUC, because a suitable range of the false positive rate can be focused according to various clinical situations. However, existing pAUC-based methods only handle a few markers and do not take nonlinear combination of markers into consideration.     

ROC曲线分析在评价入侵物种分布模型中的应用

生态位模型（ecological niche models,ENMs）已广泛应用于物种潜在分布区预测,ENMs的应用也为外来入侵物种的风险分析提供了重要的定量化分析工具,但如何评价不同模型之间的预测效果成了当今研究的热点问题。本文介绍了受试者工作特征（ROC）曲线分析在评价不同生态位模型预测效果中的应用原理和分析方法,并以一种植物病原线虫-相似穿孔线虫（Radopholus similis）为例,应用ROC曲线分析法对其5种模型（BIOCLIM,CLIMEX,DOMAIN,GARP,MAXENT）的预测结果进行了比较分析。5种模型的ROC曲线下面积AUC（Area Under Curve）值分别为0.810,0.758,0.921,0.903和0.950,以MAXENT模型的AUC值最大,表明其预测效果最好;方差分析结果表明,除GARP与DOMAIN模型之间AUC值差异不显著外,其余各模型之间差异显著。     

Time‐dependent classification accuracy curve under marker‐dependent sampling

Evaluating the classification accuracy of a candidate biomarker signaling the onset of disease or disease status is essential for medical decision making. A good biomarker would accurately identify the patients who are likely to progress or die at a particular time in the future or who are in urgent need for active treatments. To assess the performance of a candidate biomarker, the receiver operating characteristic (ROC) curve and the area under the ROC curve (AUC) are commonly used. In many cases, the standard simple random sampling (SRS) design used for biomarker validation studies is costly and inefficient. In order to improve the efficiency and reduce the cost of biomarker validation, marker‐dependent sampling (MDS) may be used. In a MDS design, the selection of patients to assess true survival time is dependent on the result of a biomarker assay. In this article, we introduce a nonparametric estimator for time‐dependent AUC under a MDS design. The consistency and the asymptotic normality of the proposed estimator is established. Simulation shows the unbiasedness of the proposed estimator and a significant efficiency gain of the MDS design over the SRS design.     

ROCS: receiver operating characteristic surface for class-skewed high-throughput data

The receiver operating characteristic (ROC) curve is an important tool to gauge the performance of classifiers. In certain situations of high-throughput data analysis, the data is heavily class-skewed, i.e. most features tested belong to the true negative class. In such cases, only a small portion of the ROC curve is relevant in practical terms, rendering the ROC curve and its area under the curve (AUC) insufficient for the purpose of judging classifier performance. Here we define an ROC surface (ROCS) using true positive rate (TPR), false positive rate (FPR), and true discovery rate (TDR). The ROC surface, together with the associated quantities, volume under the surface (VUS) and FDR-controlled area under the ROC curve (FCAUC), provide a useful approach for gauging classifier performance on class-skewed high-throughput data. The implementation as an R package is available at http://userwww.service.emory.edu/~tyu8/ROCS/.     

Estimating the area under the ROC curve when transporting a prediction model to a target population

We propose methods for estimating the area under the receiver operating characteristic (ROC) curve (AUC) of a prediction model in a target population that differs from the source population that provided the data used for original model development. If covariates that are associated with model performance, as measured by the AUC, have a different distribution in the source and target populations, then AUC estimators that only use data from the source population will not reflect model performance in the target population. Here, we provide identification results for the AUC in the target population when outcome and covariate data are available from the sample of the source population, but only covariate data are available from the sample of the target population. In this setting, we propose three estimators for the AUC in the target population and show that they are consistent and asymptotically normal. We evaluate the finite-sample performance of the estimators using simulations and use them to estimate the AUC in a nationally representative target population from the National Health and Nutrition Examination Survey for a lung cancer risk prediction model developed using source population data from the National Lung Screening Trial.     

Secreted ERBB3 isoforms are serum markers for early hepatoma in patients with chronic hepatitis and cirrhosis

Most hepatocellular carcinoma (HCC) is generated from chronic hepatitis and cirrhosis. To discover new markers for early HCC in patients with chronic hepatitis and cirrhosis, we initiated our search in the interstitial fluid of tumor (TIF) via differential gel electrophoresis and antibody arrays and identified secreted ERBB3 isoforms (sERBB3). The performance of serum sERBB3 in diagnosis of HCC was analyzed using receiver operating characteristic curves (ROC). The serum sERBB3 level was significantly higher in HCC than in cirrhosis (p < 0.001) and chronic hepatitis (p < 0.001). The accuracy of serum sERBB3 in detection of HCC was further validated in two independent sets of patients. In discrimination of early HCC from chronic hepatitis or cirrhosis, serum sERBB3 had a better performance than alpha-fetoprotein (AFP) (areas under ROC [AUC]: sERBB3 vs AFP = 93.1 vs 81.0% from chronic hepatitis and 70.9 vs 62.7% from cirrhosis). Combination of sERBB3 and AFP further improved the accuracy in detection of early HCC from chronic hepatitis (AUC = 97.1%) or cirrhosis (AUC = 77.5%). Higher serum sERBB3 levels were associated with portal-vein invasion and extrahepatic metastasis of HCC (p = 0.017). Therefore, sERBB3 are serum markers for early HCC in patients with chronic hepatitis and cirrhosis.     

StAR: a simple tool for the statistical comparison of ROC curves

Background  

As in many different areas of science and technology, most important problems in bioinformatics rely on the proper development and assessment of binary classifiers. A generalized assessment of the performance of binary classifiers is typically carried out through the analysis of their receiver operating characteristic (ROC) curves. The area under the ROC curve (AUC) constitutes a popular indicator of the performance of a binary classifier. However, the assessment of the statistical significance of the difference between any two classifiers based on this measure is not a straightforward task, since not many freely available tools exist. Most existing software is either not free, difficult to use or not easy to automate when a comparative assessment of the performance of many binary classifiers is intended. This constitutes the typical scenario for the optimization of parameters when developing new classifiers and also for their performance validation through the comparison to previous art.     

Threshold-dependence as a desirable attribute for discrimination assessment: implications for the evaluation of species distribution models

Species distribution modelling has become a common approach in ecology in the last decades. As in any modelling exercise, evaluation of the predicted suitability surfaces is a key process, and the area under the receiver operating characteristic (ROC) curve (AUC) has become the most popular statistic for this purpose. A close covariation between the AUC and threshold-dependent discrimination measures (sensitivity Se and specificity Sp) raises into question the advantage of the threshold-independence of the AUC. In this study, the relationship between the AUC and several threshold-dependent discrimination measures is characterized in detail, and the sensitivity of the pattern to variations in the shape of the ROC curve is assessed. Hypothetical suitability values, coming from normal and skew-normal distributions, were simulated for both instances of presence and absence. The flexibility of the skew-normal distribution allowed for the simulation of a wide range of ROC curve configurations. The relationship between the AUC and threshold-dependent measures was graphically assessed; independently of the ROC curve shape, a nonlinear asymptotic relationship between the AUC and Se (and Sp) was obtained after applying the threshold that makes Se = Sp. A nonlinear asymptotic relationship between the AUC and the Youden index was also reported. These results imply that the AUC does not appropriately measure changes in the discrimination of models, and it is especially incapable of distinguishing between models with high discrimination capacity. Se or Sp derived from the application of the threshold that makes them equal is a preferred measure of discrimination power. Together with the rate of false positives and negatives, and with the prevalence of the species, these statistics provide more information about the discrimination capacity of the models than the AUC.     

Receiver Operating Characteristic Analysis of Intraoperative Parathyroid Hormone Monitoring to Determine Optimum Sensitivity and Specificity: Analysis of 896 Cases

Objective: While intraoperative parathyroid hormone (IOPTH) monitoring with a ≥50% drop commonly guides the extent of exploration for primary hyperparathyroidism (pHPT), receiver operating characteristic (ROC) analysis has not been performed to determine whether other criteria yield better sensitivity and specificity. The aim of this study was to identify the optimum percent change of IOPTH following removal of the abnormal parathyroid pathology, in order to predict biochemical cure. Secondary aims were to identify patient subgroups with increased area under the ROC curve (AUC) and the need for moderated criteria.Methods: A retrospective review was performed on patients undergoing primary parathyroid surgery for sporadic pHPT between 1999 and 2010 at a tertiary center for endocrine surgery. Eight hundred and ninety-six patients with primary hyperparathyroidism were included. Multigland disease (MGD) was defined as the intraoperative detection of more than 1 enlarged hypercellular gland or persistent disease after single gland excision. ROC analysis was used to determine the value with the best performance at predicting MGD, following bilateral exploration.Results: MGD was diagnosed in 174 patients (19.4%). ROC analysis demonstrated an AUC of 0.69. An IOPTH drop of 72% was the point of optimal discrimination with a sensitivity of 55% and specificity of 76% for predicting MGD. Subgroup analysis by preoperative calcium, preoperative PTH, localization studies, or pre- and post-excision IOPTH, did not identify any factors associated with an improved AUC.Conclusion: To our knowledge, this is the first study to use ROC analysis in a large patient cohort. An IOPTH drop of 72% was found to have optimal discriminating ability. We failed to identify a subset of patients for whom there was substantial improvement in the AUC, sensitivity, or specificity.Abbreviations: AUC = area under the ROC curve; BE = bilateral neck exploration; FE = focal parathyroid exploration; IOPTH = intraoperative parathyroid hormone; MGD = multigland disease; MIBI = Tc99m-sestamibi I-123 subtraction single-photon emission computed tomography/computed tomography; pHPT = primary hyperparathyroidism; ROC = receiver operating characteristic; SGD = single gland disease; US = surgeon-performed neck ultrasound     

Evaluation of Major Online Diabetes Risk Calculators and Computerized Predictive Models

Classical paper-and-pencil based risk assessment questionnaires are often accompanied by the online versions of the questionnaire to reach a wider population. This study focuses on the loss, especially in risk estimation performance, that can be inflicted by direct transformation from the paper to online versions of risk estimation calculators by ignoring the possibilities of more complex and accurate calculations that can be performed using the online calculators. We empirically compare the risk estimation performance between four major diabetes risk calculators and two, more advanced, predictive models. National Health and Nutrition Examination Survey (NHANES) data from 1999–2012 was used to evaluate the performance of detecting diabetes and pre-diabetes.American Diabetes Association risk test achieved the best predictive performance in category of classical paper-and-pencil based tests with an Area Under the ROC Curve (AUC) of 0.699 for undiagnosed diabetes (0.662 for pre-diabetes) and 47% (47% for pre-diabetes) persons selected for screening. Our results demonstrate a significant difference in performance with additional benefits for a lower number of persons selected for screening when statistical methods are used. The best AUC overall was obtained in diabetes risk prediction using logistic regression with AUC of 0.775 (0.734) and an average 34% (48%) persons selected for screening. However, generalized boosted regression models might be a better option from the economical point of view as the number of selected persons for screening of 30% (47%) lies significantly lower for diabetes risk assessment in comparison to logistic regression (p < 0.001), with a significantly higher AUC (p < 0.001) of 0.774 (0.740) for the pre-diabetes group.Our results demonstrate a serious lack of predictive performance in four major online diabetes risk calculators. Therefore, one should take great care and consider optimizing the online versions of questionnaires that were primarily developed as classical paper questionnaires.     

Insights into the area under the receiver operating characteristic curve (AUC) as a discrimination measure in species distribution modelling

Aim The area under the receiver operating characteristic (ROC) curve (AUC) is a widely used statistic for assessing the discriminatory capacity of species distribution models. Here, I used simulated data to examine the interdependence of the AUC and classical discrimination measures (sensitivity and specificity) derived for the application of a threshold. I shall further exemplify with simulated data the implications of using the AUC to evaluate potential versus realized distribution models. Innovation After applying the threshold that makes sensitivity and specificity equal, a strong relationship between the AUC and these two measures was found. This result is corroborated with real data. On the other hand, the AUC penalizes the models that estimate potential distributions (the regions where the species could survive and reproduce due to the existence of suitable environmental conditions), and favours those that estimate realized distributions (the regions where the species actually lives). Main conclusions Firstly, the independence of the AUC from the threshold selection may be irrelevant in practice. This result also emphasizes the fact that the AUC assumes nothing about the relative costs of errors of omission and commission. However, in most real situations this premise may not be optimal. Measures derived from a contingency table for different cost ratio scenarios, together with the ROC curve, may be more informative than reporting just a single AUC value. Secondly, the AUC is only truly informative when there are true instances of absence available and the objective is the estimation of the realized distribution. When the potential distribution is the goal of the research, the AUC is not an appropriate performance measure because the weight of commission errors is much lower than that of omission errors.     

Semi-parametric area under the curve regression method for diagnostic studies with ordinal data

The classification accuracy of new diagnostic tests is based on receiver operating characteristic (ROC) curves. The area under the ROC curve (AUC) is one of the well-accepted summary measures for describing the accuracy of diagnostic tests. The AUC summary measure can vary by patient and testing characteristics. Thus, the performance of the test may be different in certain subpopulation of patients and readers. For this purpose, we propose a direct semi-parametric regression model for the non-parametric AUC measure for ordinal data while accounting for discrete and continuous covariates. The proposed method can be used to estimate the AUC value under degenerate data where certain rating categories are not observed. We will discuss the non-standard asymptotic theory, since the estimating functions were based on cross-correlated random variables. Simulation studies based on different classification models showed that the proposed model worked reasonably well with small percent bias and percent mean-squared error. The proposed method was applied to the prostate cancer study to estimate the AUC for four readers, and the carotid vessel study with age, gender, history of previous stroke, and total number of risk factors as covariates, to estimate the accuracy of the diagnostic test in the presence of subject-level covariates.     

A Bayesian hierarchical non-linear regression model in receiver operating characteristic analysis of clustered continuous diagnostic data

Receiver operating characteristic (ROC) analysis is a useful evaluative method of diagnostic accuracy. A Bayesian hierarchical nonlinear regression model for ROC analysis was developed. A validation analysis of diagnostic accuracy was conducted using prospective multi-center clinical trial prostate cancer biopsy data collected from three participating centers. The gold standard was based on radical prostatectomy to determine local and advanced disease. To evaluate the diagnostic performance of PSA level at fixed levels of Gleason score, a normality transformation was applied to the outcome data. A hierarchical regression analysis incorporating the effects of cluster (clinical center) and cancer risk (low, intermediate, and high) was performed, and the area under the ROC curve (AUC) was estimated.     

Five genes as a novel signature for predicting the prognosis of patients with laryngeal cancer

In this study, we purpose to investigate a novel five-gene signature for predicting the prognosis of patients with laryngeal cancer. The laryngeal cancer datasets were obtained from The Cancer Genome Atlas (TCGA). Both univariate and multivariate Cox regression analysis was applied to screening for prognostic differential expressed genes (DEGs), and a novel gene signature was obtained. The performance of this Cox regression model was tested by receiver operating characteristic (ROC) curves and area under the curve (AUC). Further survival analysis for each of the five genes was carried out through the Kaplan-Meier curve and Log-rank test. Totally, 622 DEGs were screened from the TCGA datasets in this study. We construct a five-gene signature through Cox survival analysis. Patients were divided into low- and high-risk groups depending on the median risk score, and a significant difference of the 5-year overall survival was found between these two groups (P < .05). ROC curves verified that this five-gene signature had good performance to predict the prognosis of laryngeal cancer (AUC = 0.862, P < .05). In conclusion, the five-gene signature consist of EMP1, HOXB9, DPY19L2P1, MMP1, and KLHDC7B might be applied as an independent prognosis predictor of laryngeal cancer.