A Bayesian hierarchical non-linear regression model in receiver operating characteristic analysis of clustered continuous diagnostic data

Receiver operating characteristic (ROC) analysis is a useful evaluative method of diagnostic accuracy. A Bayesian hierarchical nonlinear regression model for ROC analysis was developed. A validation analysis of diagnostic accuracy was conducted using prospective multi-center clinical trial prostate cancer biopsy data collected from three participating centers. The gold standard was based on radical prostatectomy to determine local and advanced disease. To evaluate the diagnostic performance of PSA level at fixed levels of Gleason score, a normality transformation was applied to the outcome data. A hierarchical regression analysis incorporating the effects of cluster (clinical center) and cancer risk (low, intermediate, and high) was performed, and the area under the ROC curve (AUC) was estimated.     

Validity of MAST-CLA for diagnosis of arthropod allergy using receiver operating characteristic (ROC) analysis

Many allergists are currently focusing on the development of new diagnostic tools, and are attempting to improve both the sensitivity and specificity. A multiple allergen simultaneous test-chemiluminescent assay (MASTCLA) is one of the most popular diagnostic tools used in the Republic of Korea. However, there remains controversy among allergists with regard to the cut-off point for a positive result. The present study was conducted in order to determine the validity of MAST-CLA as compared with that of the skin prick test, with particular emphasis on arthropod allergens, on the basis of percentage agreement rates and kappa-values, and also to suggest the optimal positive cutoff points using receiver operating characteristic (ROC) curves. The study was conducted with 97 subjects (54 men, 43 women). Optimal individual cut-off points were calculated as follows; class II for Dermatophagoides farinae, class I for Dermatophagoides pteronyssinus, and trace for a cockroach mix. These findings suggest that attempting to apply optimal individual cut-off points will be a good way of improving diagnostic tests, particularly MAST-CLA.     

Combining predictors for classification using the area under the receiver operating characteristic curve

No single biomarker for cancer is considered adequately sensitive and specific for cancer screening. It is expected that the results of multiple markers will need to be combined in order to yield adequately accurate classification. Typically, the objective function that is optimized for combining markers is the likelihood function. In this article, we consider an alternative objective function-the area under the empirical receiver operating characteristic curve (AUC). We note that it yields consistent estimates of parameters in a generalized linear model for the risk score but does not require specifying the link function. Like logistic regression, it yields consistent estimation with case-control or cohort data. Simulation studies suggest that AUC-based classification scores have performance comparable with logistic likelihood-based scores when the logistic regression model holds. Analysis of data from a proteomics biomarker study shows that performance can be far superior to logistic regression derived scores when the logistic regression model does not hold. Model fitting by maximizing the AUC rather than the likelihood should be considered when the goal is to derive a marker combination score for classification or prediction.     

A permutation test to compare receiver operating characteristic curves

We developed a permutation test in our earlier paper (Venkatraman and Begg, 1996, Biometrika 83, 835-848) to test the equality of receiver operating characteristic curves based on continuous paired data. Here we extend the underlying concepts to develop a permutation test for continuous unpaired data, and we study its properties through simulations.     

Adjusting for covariate effects on classification accuracy using the covariate-adjusted receiver operating characteristic curve

Recent scientific and technological innovations have produced an abundance of potential markers that are being investigated for their use in disease screening and diagnosis. In evaluating these markers, it is often necessary to account for covariates associated with the marker of interest. Covariates may include subject characteristics, expertise of the test operator, test procedures or aspects of specimen handling. In this paper, we propose the covariate-adjusted receiver operating characteristic curve, a measure of covariate-adjusted classification accuracy. Nonparametric and semiparametric estimators are proposed, asymptotic distribution theory is provided and finite sample performance is investigated. For illustration we characterize the age-adjusted discriminatory accuracy of prostate-specific antigen as a biomarker for prostate cancer.     

Nonparametric Bayesian estimation of the three-way receiver operating characteristic surface

We describe a nonparametric Bayesian approach for estimating the three-way ROC surface based on mixtures of finite Polya trees (MFPT) priors. Mixtures of finite Polya trees are robust models that can handle nonstandard features in the data. We address the difficulties in modeling continuous diagnostic data with skewness, multimodality, or other nonstandard features, and how parametric approaches can lead to misleading results in such cases. Robust, data-driven inference for the ROC surface and for the volume under the ROC surface is obtained. A simulation study is performed to assess the performance of the proposed method. Methods are applied to data from a magnetic resonance spectroscopy study on human immunodeficiency virus patients.     

Identification of broadly discriminatory tissue biomarkers of synovitis with binary and multicategory receiver operating characteristic analysis

Immunohistochemical synovial tissue biomarkers are used increasingly to classify arthropathies, study their pathogenesis, and to measure disease activity in clinical trials. We have used receiver operating characteristic (ROC) analysis to quantify the discriminatory abilities of markers for common inflammatory cells (subintimal CD15, CD68, CD3, CD20, CD38, and lining CD68), proliferating cells (Ki-67) and blood vessels (von Willebrand factor, vWF) among inflammatory (chronic septic arthritis, early arthritis and rheumatoid arthritis (RA)) and degenerative arthropathies (osteoarthritis (OA) and orthopedic arthropathies) and normal synovium. Six of the eight markers distinguished accurately between RA and the degenerative arthropathies (area under the curve (AUC) 0.91–0.97), whereas subintimal CD68 (AUC 0.92) and Ki-67 (AUC 0.87) distinguished best between OA and normal synovium. Fold differences in mean expression correlated only modestly with AUCs (r²?=?0.44). Multicategory ROC analysis ranked Ki-67, subintimal CD68, and CD15 as discriminating best among all six sample groups, and thus identified them as the most broadly applicable markers.     

Optimal weight in estimating and comparing areas under the receiver operating characteristic curve using longitudinal data

In the setting of longitudinal study, subjects are followed for the occurrence of some dichotomous outcome. In many of these studies, some markers are also obtained repeatedly during the study period. Emir et al. introduced a non-parametric approach to the estimation of the area under the ROC curve of a repeated marker. Their non-parametric estimate involves assigning a weight to each subject. There are two weighting schemes suggested in their paper: one for the case when within-patient correlation is low, and the other for the case when within-subject correlation is high. However, it is not clear how to assign weights to marker measurements when within-patient correlation is modest. In this paper, we consider the optimal weights that minimize the variance of the estimate of the area under the ROC curve (AUC) of a repeated marker, as well as the optimal weights that minimize the variance of the AUC difference between two repeated markers. Our results in this paper show that the optimal weights depend not only on the within-patient control--case correlation in the longitudinal data, but also on the proportion of subjects that become cases. More importantly, we show that the loss of efficiency by using the two weighting schemes suggested by Emir et al. instead of our optimal weights can be severe when there is a large within-subject control--case correlation and the proportion of subjects that become cases is small, which is often the case in longitudinal study settings.     

ROCS: receiver operating characteristic surface for class-skewed high-throughput data

The receiver operating characteristic (ROC) curve is an important tool to gauge the performance of classifiers. In certain situations of high-throughput data analysis, the data is heavily class-skewed, i.e. most features tested belong to the true negative class. In such cases, only a small portion of the ROC curve is relevant in practical terms, rendering the ROC curve and its area under the curve (AUC) insufficient for the purpose of judging classifier performance. Here we define an ROC surface (ROCS) using true positive rate (TPR), false positive rate (FPR), and true discovery rate (TDR). The ROC surface, together with the associated quantities, volume under the surface (VUS) and FDR-controlled area under the ROC curve (FCAUC), provide a useful approach for gauging classifier performance on class-skewed high-throughput data. The implementation as an R package is available at http://userwww.service.emory.edu/~tyu8/ROCS/.     

Prediction-based structured variable selection through the receiver operating characteristic curves

In many clinical settings, a commonly encountered problem is to assess accuracy of a screening test for early detection of a disease. In these applications, predictive performance of the test is of interest. Variable selection may be useful in designing a medical test. An example is a research study conducted to design a new screening test by selecting variables from an existing screener with a hierarchical structure among variables: there are several root questions followed by their stem questions. The stem questions will only be asked after a subject has answered the root question. It is therefore unreasonable to select a model that only contains stem variables but not its root variable. In this work, we propose methods to perform variable selection with structured variables when predictive accuracy of a diagnostic test is the main concern of the analysis. We take a linear combination of individual variables to form a combined test. We then maximize a direct summary measure of the predictive performance of the test, the area under a receiver operating characteristic curve (AUC of an ROC), subject to a penalty function to control for overfitting. Since maximizing empirical AUC of the ROC of a combined test is a complicated nonconvex problem (Pepe, Cai, and Longton, 2006, Biometrics62, 221-229), we explore the connection between the empirical AUC and a support vector machine (SVM). We cast the problem of maximizing predictive performance of a combined test as a penalized SVM problem and apply a reparametrization to impose the hierarchical structure among variables. We also describe a penalized logistic regression variable selection procedure for structured variables and compare it with the ROC-based approaches. We use simulation studies based on real data to examine performance of the proposed methods. Finally we apply developed methods to design a structured screener to be used in primary care clinics to refer potentially psychotic patients for further specialty diagnostics and treatment.     

A marginal model approach for analysis of multi-reader multi-test receiver operating characteristic (ROC) data

The receiver operating characteristic curve is a popular tool to characterize the capabilities of diagnostic tests with continuous or ordinal responses. One common design for assessing the accuracy of diagnostic tests involves multiple readers and multiple tests, in which all readers read all test results from the same patients. This design is most commonly used in a radiology setting, where the results of diagnostic tests depend on a radiologist's subjective interpretation. The most widely used approach for analyzing data from such a study is the Dorfman-Berbaum-Metz (DBM) method (Dorfman et al., 1992) which utilizes a standard analysis of variance (ANOVA) model for the jackknife pseudovalues of the area under the ROC curves (AUCs). Although the DBM method has performed well in published simulation studies, there is no clear theoretical basis for this approach. In this paper, focusing on continuous outcomes, we investigate its theoretical basis. Our result indicates that the DBM method does not satisfy the regular assumptions for standard ANOVA models, and thus might lead to erroneous inference. We then propose a marginal model approach based on the AUCs which can adjust for covariates as well. Consistent and asymptotically normal estimators are derived for regression coefficients. We compare our approach with the DBM method via simulation and by an application to data from a breast cancer study. The simulation results show that both our method and the DBM method perform well when the accuracy of tests under the study is the same and that our method outperforms the DBM method for inference on individual AUCs when the accuracy of tests is not the same. The marginal model approach can be easily extended to ordinal outcomes.     

The use of receiver operating characteristic (ROC) curves analysis in the evaluation of the diagnostic efficiency of serum pseudouridine as a tumor marker

The biochemical parameters used in this study were: (1) serum pseudouridine, expressed as nmols/mL; (2) pseudouridine index, expressed as mol to mol ratio of serum pseudouridine versus serum creatinine concentration. The receiver operating characteristic (ROC) analysis has been used to exemplify the selection of discriminant values or "cut-off points" to maximize the diagnostic utility of a biochemical tumor marker, serum pseudouridine. This marker has been used in a variety of group population samples, i.e., normal subjects, subjects affected by several nonneoplastic diseases, subjects with neoplastic disorders in less advanced or more advanced stages, and finally in a sample population of patients affected by lymphomas and leukemias of different types. An analysis of the relative ROC curves allowed the selection of cut-off values that maximize the diagnostic efficiency or, alternatively, the diagnostic sensitivity or the diagnostic specificity for pseudouridine parameters, and has allowed the comparison of the two tests to answer the same clinical question.     

Plotting receiver operating characteristic and precision–recall curves from presence and background data

1. The receiver operating characteristic (ROC) and precision–recall (PR) plots have been widely used to evaluate the performance of species distribution models. Plotting the ROC/PR curves requires a traditional test set with both presence and absence data (namely PA approach), but species absence data are usually not available in reality. Plotting the ROC/PR curves from presence‐only data while treating background data as pseudo absence data (namely PO approach) may provide misleading results.
2. In this study, we propose a new approach to calibrate the ROC/PR curves from presence and background data with user‐provided information on a constant c, namely PB approach. Here, c defines the probability that species occurrence is detected (labeled), and an estimate of c can also be derived from the PB‐based ROC/PR plots given that a model with good ability of discrimination is available. We used five virtual species and a real aerial photography to test the effectiveness of the proposed PB‐based ROC/PR plots. Different models (or classifiers) were trained from presence and background data with various sample sizes. The ROC/PR curves plotted by PA approach were used to benchmark the curves plotted by PO and PB approaches.
3. Experimental results show that the curves and areas under curves by PB approach are more similar to that by PA approach as compared with PO approach. The PB‐based ROC/PR plots also provide highly accurate estimations of c in our experiment.
4. We conclude that the proposed PB‐based ROC/PR plots can provide valuable complements to the existing model assessment methods, and they also provide an additional way to estimate the constant c (or species prevalence) from presence and background data.

     

A method to measure predictive ability of an injury risk curve using an observation-adjusted area under the receiver operating characteristic curve

Area under the receiver operating characteristic curve (AROC) is commonly used to choose a biomechanical metric from which to construct an injury risk curve (IRC). However, AROC may not handle censored datasets adequately. Survival analysis creates robust estimates of IRCs which accommodate censored data. We present an observation-adjusted ROC (oaROC) which uses the survival-based IRC to estimate the AROC. We verified and evaluated this method using simulated datasets of different censoring statuses and sample sizes. For a dataset with 1000 left and right censored observations, the median AROC closely approached the oaROC_True, or the oaROC calculated using an assumed “true” IRC, differing by a fraction of a percent, 0.1%. Using simulated datasets with various censoring, we found that oaROC converged onto oaROC_True in all cases. For datasets with right and non-censored observations, AROC did not converge onto oaROC_True. oaROC for datasets with only non-censored observations converged the fastest, and for a dataset with 10 observations, the median oaROC differed from oaROC_True by 2.74% while the corresponding median AROC with left and right censored data differed from oaROC_True by 9.74%. We also calculated the AROC and oaROC for a published side impact dataset, and differences between the two methods ranged between −24.08% and 24.55% depending on metric. Overall, when compared with AROC, we found oaROC performs equivalently for doubly censored data, better for non-censored data, and can accommodate more types of data than AROC. While more validation is needed, the results indicate that oaROC is a viable alternative which can be incorporated into the metric selection process for IRCs.