Time-dependent ROC curves for censored survival data and a diagnostic marker

ROC curves are a popular method for displaying sensitivity and specificity of a continuous diagnostic marker, X, for a binary disease variable, D. However, many disease outcomes are time dependent, D(t), and ROC curves that vary as a function of time may be more appropriate. A common example of a time-dependent variable is vital status, where D(t) = 1 if a patient has died prior to time t and zero otherwise. We propose summarizing the discrimination potential of a marker X, measured at baseline (t = 0), by calculating ROC curves for cumulative disease or death incidence by time t, which we denote as ROC(t). A typical complexity with survival data is that observations may be censored. Two ROC curve estimators are proposed that can accommodate censored data. A simple estimator is based on using the Kaplan-Meier estimator for each possible subset X > c. However, this estimator does not guarantee the necessary condition that sensitivity and specificity are monotone in X. An alternative estimator that does guarantee monotonicity is based on a nearest neighbor estimator for the bivariate distribution function of (X, T), where T represents survival time (Akritas, M. J., 1994, Annals of Statistics 22, 1299-1327). We present an example where ROC(t) is used to compare a standard and a modified flow cytometry measurement for predicting survival after detection of breast cancer and an example where the ROC(t) curve displays the impact of modifying eligibility criteria for sample size and power in HIV prevention trials.     

Translational biomarker discovery in clinical metabolomics: an introductory tutorial

Metabolomics is increasingly being applied towards the identification of biomarkers for disease diagnosis, prognosis and risk prediction. Unfortunately among the many published metabolomic studies focusing on biomarker discovery, there is very little consistency and relatively little rigor in how researchers select, assess or report their candidate biomarkers. In particular, few studies report any measure of sensitivity, specificity, or provide receiver operator characteristic (ROC) curves with associated confidence intervals. Even fewer studies explicitly describe or release the biomarker model used to generate their ROC curves. This is surprising given that for biomarker studies in most other biomedical fields, ROC curve analysis is generally considered the standard method for performance assessment. Because the ultimate goal of biomarker discovery is the translation of those biomarkers to clinical practice, it is clear that the metabolomics community needs to start “speaking the same language” in terms of biomarker analysis and reporting-especially if it wants to see metabolite markers being routinely used in the clinic. In this tutorial, we will first introduce the concept of ROC curves and describe their use in single biomarker analysis for clinical chemistry. This includes the construction of ROC curves, understanding the meaning of area under ROC curves (AUC) and partial AUC, as well as the calculation of confidence intervals. The second part of the tutorial focuses on biomarker analyses within the context of metabolomics. This section describes different statistical and machine learning strategies that can be used to create multi-metabolite biomarker models and explains how these models can be assessed using ROC curves. In the third part of the tutorial we discuss common issues and potential pitfalls associated with different analysis methods and provide readers with a list of nine recommendations for biomarker analysis and reporting. To help readers test, visualize and explore the concepts presented in this tutorial, we also introduce a web-based tool called ROCCET (ROC Curve Explorer & Tester, http://www.roccet.ca). ROCCET was originally developed as a teaching aid but it can also serve as a training and testing resource to assist metabolomics researchers build biomarker models and conduct a range of common ROC curve analyses for biomarker studies.     

ROC-guided survival trees and ensembles

Tree-based methods are popular nonparametric tools in studying time-to-event outcomes. In this article, we introduce a novel framework for survival trees and ensembles, where the trees partition the dynamic survivor population and can handle time-dependent covariates. Using the idea of randomized tests, we develop generalized time-dependent receiver operating characteristic (ROC) curves for evaluating the performance of survival trees. The tree-building algorithm is guided by decision-theoretic criteria based on ROC, targeting specifically for prediction accuracy. To address the instability issue of a single tree, we propose a novel ensemble procedure based on averaging martingale estimating equations, which is different from existing methods that average the predicted survival or cumulative hazard functions from individual trees. Extensive simulation studies are conducted to examine the performance of the proposed methods. We apply the methods to a study on AIDS for illustration.     

On the use of partial area under the ROC curve for comparison of two diagnostic tests

Evaluation of diagnostic performance is typically based on the receiver operating characteristic (ROC) curve and the area under the curve (AUC) as its summary index. The partial area under the curve (pAUC) is an alternative index focusing on the range of practical/clinical relevance. One of the problems preventing more frequent use of the pAUC is the perceived loss of efficiency in cases of noncrossing ROC curves. In this paper, we investigated statistical properties of comparisons of two correlated pAUCs. We demonstrated that outside of the classic model there are practically reasonable ROC types for which comparisons of noncrossing concave curves would be more powerful when based on a part of the curve rather than the entire curve. We argue that this phenomenon stems in part from the exclusion of noninformative parts of the ROC curves that resemble straight‐lines. We conducted extensive simulation studies in families of binormal, straight‐line, and bigamma ROC curves. We demonstrated that comparison of pAUCs is statistically more powerful than comparison of full AUCs when ROC curves are close to a “straight line”. For less flat binormal ROC curves an increase in the integration range often leads to a disproportional increase in pAUCs’ difference, thereby contributing to an increase in statistical power. Thus, efficiency of differences in pAUCs of noncrossing ROC curves depends on the shape of the curves, and for families of ROC curves that are nearly straight‐line shaped, such as bigamma ROC curves, there are multiple practical scenarios in which comparisons of pAUCs are preferable.     

Semiparametric estimation of the covariate-specific ROC curve in presence of ignorable verification bias

Covariate-specific receiver operating characteristic (ROC) curves are often used to evaluate the classification accuracy of a medical diagnostic test or a biomarker, when the accuracy of the test is associated with certain covariates. In many large-scale screening tests, the gold standard is subject to missingness due to high cost or harmfulness to the patient. In this article, we propose a semiparametric estimation of the covariate-specific ROC curves with a partial missing gold standard. A location-scale model is constructed for the test result to model the covariates' effect, but the residual distributions are left unspecified. Thus the baseline and link functions of the ROC curve both have flexible shapes. With the gold standard missing at random (MAR) assumption, we consider weighted estimating equations for the location-scale parameters, and weighted kernel estimating equations for the residual distributions. Three ROC curve estimators are proposed and compared, namely, imputation-based, inverse probability weighted, and doubly robust estimators. We derive the asymptotic normality of the estimated ROC curve, as well as the analytical form of the standard error estimator. The proposed method is motivated and applied to the data in an Alzheimer's disease research.     

ROC Curve Analysis in the Presence of Imperfect Reference Standards

The receiver operating characteristic (ROC) curve is an important tool for the evaluation and comparison of predictive models when the outcome is binary. If the class membership of the outcomes is known, ROC can be constructed for a model, and the ROC with greater area under the curve indicates better performance. However in practice, imperfect reference standards often exist, in which class membership of every data point is not fully determined. This situation is especially prevalent in high-throughput biomedical data because obtaining perfect reference standards for all data points is either too costly or technically impractical. To construct ROC curves for these data, the common practice is to either ignore the uncertainties in references or remove data points with high uncertainties. Such approaches may cause bias to the ROC curves and generate misleading results in method evaluation. Here we present a framework to incorporate membership uncertainties into the construction of ROC curve, termed the expected ROC or “eROC” curve. We develop an efficient procedure for the estimation of eROC curve. The advantages of using eROC are demonstrated using simulated and real data.     

Plotting receiver operating characteristic and precision–recall curves from presence and background data

1. The receiver operating characteristic (ROC) and precision–recall (PR) plots have been widely used to evaluate the performance of species distribution models. Plotting the ROC/PR curves requires a traditional test set with both presence and absence data (namely PA approach), but species absence data are usually not available in reality. Plotting the ROC/PR curves from presence‐only data while treating background data as pseudo absence data (namely PO approach) may provide misleading results.
2. In this study, we propose a new approach to calibrate the ROC/PR curves from presence and background data with user‐provided information on a constant c, namely PB approach. Here, c defines the probability that species occurrence is detected (labeled), and an estimate of c can also be derived from the PB‐based ROC/PR plots given that a model with good ability of discrimination is available. We used five virtual species and a real aerial photography to test the effectiveness of the proposed PB‐based ROC/PR plots. Different models (or classifiers) were trained from presence and background data with various sample sizes. The ROC/PR curves plotted by PA approach were used to benchmark the curves plotted by PO and PB approaches.
3. Experimental results show that the curves and areas under curves by PB approach are more similar to that by PA approach as compared with PO approach. The PB‐based ROC/PR plots also provide highly accurate estimations of c in our experiment.
4. We conclude that the proposed PB‐based ROC/PR plots can provide valuable complements to the existing model assessment methods, and they also provide an additional way to estimate the constant c (or species prevalence) from presence and background data.

     

An interpretation for the ROC curve and inference using GLM procedures

The accuracy of a medical diagnostic test is often summarized in a receiver operating characteristic (ROC) curve. This paper puts forth an interpretation for each point on the ROC curve as being a conditional probability of a test result from a random diseased subject exceeding that from a random nondiseased subject. This interpretation gives rise to new methods for making inference about ROC curves. It is shown that inference can be achieved with binary regression techniques applied to indicator variables constructed from pairs of test results, one component of the pair being from a diseased subject and the other from a nondiseased subject. Within the generalized linear model (GLM) binary regression framework, ROC curves can be estimated, and we highlight a new semiparametric estimator. Covariate effects can also be evaluated with the GLM models. The methodology is applied to a pancreatic cancer dataset where we use the regression framework to compare two different serum biomarkers. Asymptotic distribution theory is developed to facilitate inference and to provide insight into factors influencing variability of estimated model parameters.     

Robust Estimation of Area Under ROC Curve Using Auxiliary Variables in the Presence of Missing Biomarker Values

Summary In medical research, the receiver operating characteristic (ROC) curves can be used to evaluate the performance of biomarkers for diagnosing diseases or predicting the risk of developing a disease in the future. The area under the ROC curve (ROC AUC), as a summary measure of ROC curves, is widely utilized, especially when comparing multiple ROC curves. In observational studies, the estimation of the AUC is often complicated by the presence of missing biomarker values, which means that the existing estimators of the AUC are potentially biased. In this article, we develop robust statistical methods for estimating the ROC AUC and the proposed methods use information from auxiliary variables that are potentially predictive of the missingness of the biomarkers or the missing biomarker values. We are particularly interested in auxiliary variables that are predictive of the missing biomarker values. In the case of missing at random (MAR), that is, missingness of biomarker values only depends on the observed data, our estimators have the attractive feature of being consistent if one correctly specifies, conditional on auxiliary variables and disease status, either the model for the probabilities of being missing or the model for the biomarker values. In the case of missing not at random (MNAR), that is, missingness may depend on the unobserved biomarker values, we propose a sensitivity analysis to assess the impact of MNAR on the estimation of the ROC AUC. The asymptotic properties of the proposed estimators are studied and their finite‐sample behaviors are evaluated in simulation studies. The methods are further illustrated using data from a study of maternal depression during pregnancy.     

Least squares regression methods for clustered ROC data with discrete covariates

The receiver operating characteristic (ROC) curve is a popular tool to evaluate and compare the accuracy of diagnostic tests to distinguish the diseased group from the nondiseased group when test results from tests are continuous or ordinal. A complicated data setting occurs when multiple tests are measured on abnormal and normal locations from the same subject and the measurements are clustered within the subject. Although least squares regression methods can be used for the estimation of ROC curve from correlated data, how to develop the least squares methods to estimate the ROC curve from the clustered data has not been studied. Also, the statistical properties of the least squares methods under the clustering setting are unknown. In this article, we develop the least squares ROC methods to allow the baseline and link functions to differ, and more importantly, to accommodate clustered data with discrete covariates. The methods can generate smooth ROC curves that satisfy the inherent continuous property of the true underlying curve. The least squares methods are shown to be more efficient than the existing nonparametric ROC methods under appropriate model assumptions in simulation studies. We apply the methods to a real example in the detection of glaucomatous deterioration. We also derive the asymptotic properties of the proposed methods.     

Prediction of multi-criteria optimization (MCO) parameter efficiency in volumetric modulated arc therapy (VMAT) treatment planning using machine learning (ML)

PurposeTo predict the impact of optimization parameter changes on dosimetric plan quality criteria in multi-criteria optimized volumetric-modulated-arc therapy (VMAT) planning prior to optimization using machine learning (ML).MethodsA data base comprising a total of 21,266 VMAT treatment plans for 44 cranial and 18 spinal patient geometries was generated. The underlying optimization algorithm is governed by three highly composite parameters which model a combination of important aspects of the solution. Patient geometries were parametrized via volume- and shape properties of the voxel objects and overlap-volume histograms (OVH) of the planning-target-volume (PTV) and a relevant organ-at-risk (OAR). The impact of changes in one of the three optimization parameters on the maximally achievable value range of five dosimetric properties of the resulting dose distributions was studied. To predict the extent of this impact based on patient geometry, treatment site, and current parameter settings prior to optimization, three different ML-models were trained and tested. Precision-recall curves, as well as the area-under-curve (AUC) of the resulting receiver-operator-characteristic (ROC) curves were analyzed for model assessment.ResultsSuccessful identification of parameter regions resulting in a high variability of dosimetric plan properties depended on the choice of geometry features, the treatment indication and the plan property under investigation. AUC values between 0.82 and 0.99 could be achieved. The best average-precision (AP) values obtained from the corresponding precision/recall curves ranged from 0.71 to 0.99.ConclusionsMachine learning models trained on a database of pre-optimized treatment plans can help finding relevant optimization parameter ranges prior to optimization.     

Construction and Comparison of Two Receiver Operating Characteristic Curves Derived from the Same Samples

Receiver operating characteristic (ROC) curves are used to describe the performance of diagnostic procedures. This paper proposes a simple method for the statistical comparison of two ROC curves derived from the same set of patients and the same set of healthy subjects. Generalization to studies involving more than two screening factors is straightforward. This method does not require the calculation of variances of the areas or difference of areas under the curves.     

Bivariate Marker Measurements and ROC Analysis

Summary This article considers receiver operating characteristic (ROC) analysis for bivariate marker measurements. The research interest is to extend tools and rules from univariate marker to bivariate marker setting for evaluating predictive accuracy of markers using a tree‐based classification rule. Using an and–or classifier, an ROC function together with a weighted ROC function (WROC) and their conjugate counterparts are proposed for examining the performance of bivariate markers. The proposed functions evaluate the performance of and–or classifiers among all possible combinations of marker values, and are ideal measures for understanding the predictability of biomarkers in target population. Specific features of ROC and WROC functions and other related statistics are discussed in comparison with those familiar properties for univariate marker. Nonparametric methods are developed for estimating ROC‐related functions (partial) area under curve and concordance probability. With emphasis on average performance of markers, the proposed procedures and inferential results are useful for evaluating marker predictability based on a single or bivariate marker (or test) measurements with different choices of markers, and for evaluating different and–or combinations in classifiers. The inferential results developed in this article also extend to multivariate markers with a sequence of arbitrarily combined and–or classifier.     

pROC: an open-source package for R and S+ to analyze and compare ROC curves

Background  

Receiver operating characteristic (ROC) curves are useful tools to evaluate classifiers in biomedical and bioinformatics applications. However, conclusions are often reached through inconsistent use or insufficient statistical analysis. To support researchers in their ROC curves analysis we developed pROC, a package for R and S+ that contains a set of tools displaying, analyzing, smoothing and comparing ROC curves in a user-friendly, object-oriented and flexible interface.     

Using ROC Curves to Choose Minimally Important Change Thresholds when Sensitivity and Specificity Are Valued Equally: The Forgotten Lesson of Pythagoras. Theoretical Considerations and an Example Application of Change in Health Status

BackgroundReceiver Operator Characteristic (ROC) curves are being used to identify Minimally Important Change (MIC) thresholds on scales that measure a change in health status. In quasi-continuous patient reported outcome measures, such as those that measure changes in chronic diseases with variable clinical trajectories, sensitivity and specificity are often valued equally. Notwithstanding methodologists agreeing that these should be valued equally, different approaches have been taken to estimating MIC thresholds using ROC curves.MethodsUsing graphical methods, hypothetical data, and data from a large randomised controlled trial of manual therapy for low back pain, we compared two existing approaches with a new approach that is based on the addition of the sums of squares of 1-sensitivity and 1-specificity.ResultsThere can be divergence in the thresholds chosen by different estimators. The cut-point selected by different estimators is dependent on the relationship between the cut-points in ROC space and the different contours described by the estimators. In particular, asymmetry and the number of possible cut-points affects threshold selection.ConclusionChoice of MIC estimator is important. Different methods for choosing cut-points can lead to materially different MIC thresholds and thus affect results of responder analyses and trial conclusions. An estimator based on the smallest sum of squares of 1-sensitivity and 1-specificity is preferable when sensitivity and specificity are valued equally. Unlike other methods currently in use, the cut-point chosen by the sum of squares method always and efficiently chooses the cut-point closest to the top-left corner of ROC space, regardless of the shape of the ROC curve.     

A nomogram model integrating LI-RADS features and radiomics based on contrast-enhanced magnetic resonance imaging for predicting microvascular invasion in hepatocellular carcinoma falling the Milan criteria

PurposeTo establish and validate a nomogram model incorporating both liver imaging reporting and data system (LI-RADS) features and contrast enhanced magnetic resonance imaging (CEMRI)-based radiomics for predicting microvascular invasion (MVI) in hepatocellular carcinoma (HCC) falling the Milan criteria.MethodsIn total, 161 patients with 165 HCCs diagnosed with MVI (n = 99) or without MVI (n = 66) were assigned to a training and a test group. MRI LI-RADS characteristics and radiomics features selected by the LASSO algorithm were used to establish the MRI and Rad-score models, respectively, and the independent features were integrated to develop the nomogram model. The predictive ability of the nomogram was evaluated with receiver operating characteristic (ROC) curves.ResultsThe risk factors associated with MVI (P<0.05) were related to larger tumor size, nonsmooth margin, mosaic architecture, corona enhancement and higher Rad-score. The areas under the ROC curve (AUCs) of the MRI feature model for predicting MVI were 0.85 (95% CI: 0.78–0.92) and 0.85 (95% CI: 0.74–0.95), and those for the Rad-score were 0.82 (95% CI: 0.73–0.90) and 0.80 (95% CI: 0.67–0.93) in the training and test groups, respectively. The nomogram presented improved AUC values of 0.87 (95% CI: 0.81–0.94) in the training group and 0.89 (95% CI: 0.81–0.98) in the test group (P<0.05) for predicting MVI. The calibration curve and decision curve analysis demonstrated that the nomogram model had high goodness-of-fit and clinical benefits.ConclusionsThe nomogram model can effectively predict MVI in patients with HCC falling within the Milan criteria and serves as a valuable imaging biomarker for facilitating individualized decision-making.     

Identifying combinations of cancer markers for further study as triggers of early intervention

In many long-term clinical trials or cohort studies, investigators repeatedly collect and store tissue or serum specimens and later test specimens from cancer cases and a random sample of controls for potential markers for cancer. An important question is what combination, if any, of the molecular markers should be studied in a future trial as a trigger for early intervention. To answer this question, we summarized the performance of various combinations using Receiver Operating Characteristic (ROC) curves, which plot true versus false positive rates. To construct the ROC curves, we proposed a new class of nonparametric algorithms which extends the ROC paradigm to multiple tests. We fit various combinations of markers to a training sample and evaluated the performance in a test sample using a target region based on a utility function. We applied the methodology to the following markers for prostate cancer, the last value of total prostate-specific antigen (PSA), the last ratio of total to free PSA, the last slope of total PSA, and the last slope of the ratio. In the test sample, the ROC curve for last total PSA was slightly closer to the target region than the ROC curve for a combination of four markers. In a separate validation sample, the ROC curve for last total PSA intersected the target region in 77% of bootstrap replications, indicating some promise for further study. We also discussed sample size calculations.     

ROC graphs for assessing the ability of a diagnostic marker to detect three disease classes with an umbrella ordering

Receiver operating characteristic (ROC) curves and the area under these curves are commonly used to assess the ability of a continuous diagnostic marker (e.g., DNA methylation markers) to correctly classify subjects as having a particular disease or not (e.g., cancer). These approaches, however, are not applicable to settings where the gold standard yields more than two disease states or classes. ROC surfaces and the volume under the surfaces have been proposed for settings with more than two disease classes. These approaches, however, do not allow one to assess the ability of a marker to differentiate two disease classes from a third disease class without requiring a monotone order for the three disease classes under study. That is, existing approaches do not accommodate an umbrella ordering of disease classes. This article proposes the construction of an ROC graph that is applicable for an umbrella ordering. Furthermore, this article proposes that a summary measure for this umbrella ROC graph can be used to summarize the classification accuracy, and corresponding variance estimates can be obtained using U-statistics theory or bootstrap methods. The proposed methods are illustrated using data from a study assessing the ability of a DNA methylation marker to correctly classify lung specimens into three histologic classes: squamous cell carcinoma, large cell carcinoma, and nontumor lung.