An interpretation for the ROC curve and inference using GLM procedures

The accuracy of a medical diagnostic test is often summarized in a receiver operating characteristic (ROC) curve. This paper puts forth an interpretation for each point on the ROC curve as being a conditional probability of a test result from a random diseased subject exceeding that from a random nondiseased subject. This interpretation gives rise to new methods for making inference about ROC curves. It is shown that inference can be achieved with binary regression techniques applied to indicator variables constructed from pairs of test results, one component of the pair being from a diseased subject and the other from a nondiseased subject. Within the generalized linear model (GLM) binary regression framework, ROC curves can be estimated, and we highlight a new semiparametric estimator. Covariate effects can also be evaluated with the GLM models. The methodology is applied to a pancreatic cancer dataset where we use the regression framework to compare two different serum biomarkers. Asymptotic distribution theory is developed to facilitate inference and to provide insight into factors influencing variability of estimated model parameters.     

The Skill Plot: a graphical technique for evaluating continuous diagnostic tests

Summary .   We introduce the Skill Plot, a method that it is directly relevant to a decision maker who must use a diagnostic test. In contrast to ROC curves, the skill curve allows easy graphical inspection of the optimal cutoff or decision rule for a diagnostic test. The skill curve and test also determine whether diagnoses based on this cutoff improve upon a naive forecast (of always present or of always absent). The skill measure makes it easy to directly compare the predictive utility of two different classifiers in an analogy to the area under the curve statistic related to ROC analysis. Finally, this article shows that the skill-based cutoff inferred from the plot is equivalent to the cutoff indicated by optimizing the posterior odds in accordance with Bayesian decision theory. A method for constructing a confidence interval for this optimal point is presented and briefly discussed.     

Distribution-free ROC analysis using binary regression techniques

Receiver operating characteristic (ROC) regression methodology is used to identify factors that affect the accuracy of medical diagnostic tests. In this paper, we consider a ROC model for which the ROC curve is a parametric function of covariates but distributions of the diagnostic test results are not specified. Covariates can be either common to all subjects or specific to those with disease. We propose a new estimation procedure based on binary indicators defined by the test result for a diseased subject exceeding various specified quantiles of the distribution of test results from non-diseased subjects with the same covariate values. This procedure is conceptually and computationally simplified relative to existing procedures. Simulation study results indicate that the approach has fairly high statistical efficiency. The new ROC regression methodology is used to evaluate childhood measurements of body mass index as a predictive marker of adult obesity.     

Combining diagnostic test results to increase accuracy

When multiple diagnostic tests are performed on an individual or multiple disease markers are available it may be possible to combine the information to diagnose disease. We consider how to choose linear combinations of markers in order to optimize diagnostic accuracy. The accuracy index to be maximized is the area or partial area under the receiver operating characteristic (ROC) curve. We propose a distribution-free rank-based approach for optimizing the area under the ROC curve and compare it with logistic regression and with classic linear discriminant analysis (LDA). It has been shown that the latter method optimizes the area under the ROC curve when test results have a multivariate normal distribution for diseased and non-diseased populations. Simulation studies suggest that the proposed non-parametric method is efficient when data are multivariate normal.The distribution-free method is generalized to a smooth distribution-free approach to: (i) accommodate some reasonable smoothness assumptions; (ii) incorporate covariate effects; and (iii) yield optimized partial areas under the ROC curve. This latter feature is particularly important since it allows one to focus on a region of the ROC curve which is of most relevance to clinical practice. Neither logistic regression nor LDA necessarily maximize partial areas. The approaches are illustrated on two cancer datasets, one involving serum antigen markers for pancreatic cancer and the other involving longitudinal prostate specific antigen data.     

Least squares regression methods for clustered ROC data with discrete covariates

The receiver operating characteristic (ROC) curve is a popular tool to evaluate and compare the accuracy of diagnostic tests to distinguish the diseased group from the nondiseased group when test results from tests are continuous or ordinal. A complicated data setting occurs when multiple tests are measured on abnormal and normal locations from the same subject and the measurements are clustered within the subject. Although least squares regression methods can be used for the estimation of ROC curve from correlated data, how to develop the least squares methods to estimate the ROC curve from the clustered data has not been studied. Also, the statistical properties of the least squares methods under the clustering setting are unknown. In this article, we develop the least squares ROC methods to allow the baseline and link functions to differ, and more importantly, to accommodate clustered data with discrete covariates. The methods can generate smooth ROC curves that satisfy the inherent continuous property of the true underlying curve. The least squares methods are shown to be more efficient than the existing nonparametric ROC methods under appropriate model assumptions in simulation studies. We apply the methods to a real example in the detection of glaucomatous deterioration. We also derive the asymptotic properties of the proposed methods.     

Estimation of the ROC Curve under Verification Bias

The ROC (receiver operating characteristic) curve is the most commonly used statistical tool for describing the discriminatory accuracy of a diagnostic test. Classical estimation of the ROC curve relies on data from a simple random sample from the target population. In practice, estimation is often complicated due to not all subjects undergoing a definitive assessment of disease status (verification). Estimation of the ROC curve based on data only from subjects with verified disease status may be badly biased. In this work we investigate the properties of the doubly robust (DR) method for estimating the ROC curve under verification bias originally developed by Rotnitzky, Faraggi and Schisterman (2006) for estimating the area under the ROC curve. The DR method can be applied for continuous scaled tests and allows for a non‐ignorable process of selection to verification. We develop the estimator's asymptotic distribution and examine its finite sample properties via a simulation study. We exemplify the DR procedure for estimation of ROC curves with data collected on patients undergoing electron beam computer tomography, a diagnostic test for calcification of the arteries.     

Confidence interval estimation of the difference between two sensitivities to the early disease stage

Although most of the statistical methods for diagnostic studies focus on disease processes with binary disease status, many diseases can be naturally classified into three ordinal diagnostic categories, that is normal, early stage, and fully diseased. For such diseases, the volume under the ROC surface (VUS) is the most commonly used index of diagnostic accuracy. Because the early disease stage is most likely the optimal time window for therapeutic intervention, the sensitivity to the early diseased stage has been suggested as another diagnostic measure. For the purpose of comparing the diagnostic abilities on early disease detection between two markers, it is of interest to estimate the confidence interval of the difference between sensitivities to the early diseased stage. In this paper, we present both parametric and non‐parametric methods for this purpose. An extensive simulation study is carried out for a variety of settings for the purpose of evaluating and comparing the performance of the proposed methods. A real example of Alzheimer's disease (AD) is analyzed using the proposed approaches.     

Estimation of the Youden Index and its associated cutoff point

The Youden Index is a frequently used summary measure of the ROC (Receiver Operating Characteristic) curve. It both, measures the effectiveness of a diagnostic marker and enables the selection of an optimal threshold value (cutoff point) for the marker. In this paper we compare several estimation procedures for the Youden Index and its associated cutoff point. These are based on (1) normal assumptions; (2) transformations to normality; (3) the empirical distribution function; (4) kernel smoothing. These are compared in terms of bias and root mean square error in a large variety of scenarios by means of an extensive simulation study. We find that the empirical method which is the most commonly used has the overall worst performance. In the estimation of the Youden Index the kernel is generally the best unless the data can be well transformed to achieve normality whereas in estimation of the optimal threshold value results are more variable.     

Regression models for convex ROC curves

The performance of a diagnostic test is summarized by its receiver operating characteristic (ROC) curve. Under quite natural assumptions about the latent variable underlying the test, the ROC curve is convex. Empirical data on a test's performance often comes in the form of observed true positive and false positive relative frequencies under varying conditions. This paper describes a family of regression models for analyzing such data. The underlying ROC curves are specified by a quality parameter delta and a shape parameter mu and are guaranteed to be convex provided delta > 1. Both the position along the ROC curve and the quality parameter delta are modeled linearly with covariates at the level of the individual. The shape parameter mu enters the model through the link functions log(p mu) - log(1 - p mu) of a binomial regression and is estimated either by search or from an appropriate constructed variate. One simple application is to the meta-analysis of independent studies of the same diagnostic test, illustrated on some data of Moses, Shapiro, and Littenberg (1993). A second application, to so-called vigilance data, is given, where ROC curves differ across subjects and modeling of the position along the ROC curve is of primary interest.     

Partial AUC estimation and regression

Accurate diagnosis of disease is a critical part of health care. New diagnostic and screening tests must be evaluated based on their abilities to discriminate diseased from nondiseased states. The partial area under the receiver operating characteristic (ROC) curve is a measure of diagnostic test accuracy. We present an interpretation of the partial area under the curve (AUC), which gives rise to a nonparametric estimator. This estimator is more robust than existing estimators, which make parametric assumptions. We show that the robustness is gained with only a moderate loss in efficiency. We describe a regression modeling framework for making inference about covariate effects on the partial AUC. Such models can refine knowledge about test accuracy. Model parameters can be estimated using binary regression methods. We use the regression framework to compare two prostate-specific antigen biomarkers and to evaluate the dependence of biomarker accuracy on the time prior to clinical diagnosis of prostate cancer.     

The optimal ratio of cases to controls for estimating the classification accuracy of a biomarker

The case-control design is frequently used to study the discriminatory accuracy of a screening or diagnostic biomarker. Yet, the appropriate ratio in which to sample cases and controls has never been determined. It is common for researchers to sample equal numbers of cases and controls, a strategy that can be optimal for studies of association. However, considerations are quite different when the biomarker is to be used for classification. In this paper, we provide an expression for the optimal case-control ratio, when the accuracy of the biomarker is quantified by the receiver operating characteristic (ROC) curve. We show how it can be integrated with choosing the overall sample size to yield an efficient study design with specified power and type-I error. We also derive the optimal case-control ratios for estimating the area under the ROC curve and the area under part of the ROC curve. Our methods are applied to a study of a new marker for adenocarcinoma in patients with Barrett's esophagus.     

Validity of MAST-CLA for diagnosis of arthropod allergy using receiver operating characteristic (ROC) analysis

Many allergists are currently focusing on the development of new diagnostic tools, and are attempting to improve both the sensitivity and specificity. A multiple allergen simultaneous test-chemiluminescent assay (MASTCLA) is one of the most popular diagnostic tools used in the Republic of Korea. However, there remains controversy among allergists with regard to the cut-off point for a positive result. The present study was conducted in order to determine the validity of MAST-CLA as compared with that of the skin prick test, with particular emphasis on arthropod allergens, on the basis of percentage agreement rates and kappa-values, and also to suggest the optimal positive cutoff points using receiver operating characteristic (ROC) curves. The study was conducted with 97 subjects (54 men, 43 women). Optimal individual cut-off points were calculated as follows; class II for Dermatophagoides farinae, class I for Dermatophagoides pteronyssinus, and trace for a cockroach mix. These findings suggest that attempting to apply optimal individual cut-off points will be a good way of improving diagnostic tests, particularly MAST-CLA.     

Semiparametric estimation of the covariate-specific ROC curve in presence of ignorable verification bias

Covariate-specific receiver operating characteristic (ROC) curves are often used to evaluate the classification accuracy of a medical diagnostic test or a biomarker, when the accuracy of the test is associated with certain covariates. In many large-scale screening tests, the gold standard is subject to missingness due to high cost or harmfulness to the patient. In this article, we propose a semiparametric estimation of the covariate-specific ROC curves with a partial missing gold standard. A location-scale model is constructed for the test result to model the covariates' effect, but the residual distributions are left unspecified. Thus the baseline and link functions of the ROC curve both have flexible shapes. With the gold standard missing at random (MAR) assumption, we consider weighted estimating equations for the location-scale parameters, and weighted kernel estimating equations for the residual distributions. Three ROC curve estimators are proposed and compared, namely, imputation-based, inverse probability weighted, and doubly robust estimators. We derive the asymptotic normality of the estimated ROC curve, as well as the analytical form of the standard error estimator. The proposed method is motivated and applied to the data in an Alzheimer's disease research.     

miR-126 在膀胱癌患者尿液中的表达及临床意义

目的:探讨miR-126在膀胱癌患者尿液中的表达与临床病理特征的关系,评估miR-126的肿瘤标志物诊断价值。方法:收集48例初发膀胱尿路上皮癌患者与32例健康对照者晨尿,提取尿液总RNA,通过实时荧光定量PCR技术检测各样本中的miR-126的表达水平,并经受试者工作曲线(ROC)分析其诊断价值。结果:膀胱癌患者尿液中的miR-126表达水平相对健康对照组明显上调(P0.01),其表达水平在不同病理级别之间存在显著差异(P均0.05),且低级别组表达水平略高于高级别组,与肿瘤大小、数目以及淋巴转移也有一定的相关性(P0.05),而与患者的年龄、性别、TNM分期等均无相关性(P0.05)。通过ROC曲线分析尿液中miR-126诊断膀胱肿瘤的曲线下面积(AUC)为0.861,当最佳切点定在7.475时,miR-126诊断膀胱肿瘤的敏感性和特异性分别为75.0%、81.2%。结论:膀胱癌患者尿液中miR-126的表达差异能够反映病情进展程度,其表达水平对膀胱肿瘤的早期诊断及病情评估具有一定的价值。     

ROC curve estimation when covariates affect the verification process

A receiver operating characteristic (ROC) curve is commonly used to measure the accuracy of a medical test. It is a plot of the true positive fraction (sensitivity) against the false positive fraction (1-specificity) for increasingly stringent positivity criterion. Bias can occur in estimation of an ROC curve if only some of the tested patients are selected for disease verification and if analysis is restricted only to the verified cases. This bias is known as verification bias. In this paper, we address the problem of correcting for verification bias in estimation of an ROC curve when the verification process and efficacy of the diagnostic test depend on covariates. Our method applies the EM algorithm to ordinal regression models to derive ML estimates for ROC curves as a function of covariates, adjusted for covariates affecting the likelihood of being verified. Asymptotic variance estimates are obtained using the observed information matrix of the observed data. These estimates are derived under the missing-at-random assumption, which means that selection for disease verification depends only on the observed data, i.e., the test result and the observed covariates. We also address the issues of model selection and model checking. Finally, we illustrate the proposed method on data from a two-phase study of dementia disorders, where selection for verification depends on the screening test result and age.     

Serum miR‐92a‐1 is a novel diagnostic biomarker for colorectal cancer

Colorectal cancer (CRC) is one of the most common cancers worldwide, with high mortality. Abnormally expressed microRNAs (miRNAs) are considered novel biomarkers in cancer diagnosis. The aim of this study was to investigate the diagnostic value of miR‐92a‐1 in patients with CRC. Serum samples were collected from 148 patients pathologically diagnosed with CRC and 68 gender‐ and age‐matched healthy volunteers. Quantitative real‐time polymerase chain reaction (qRT‐PCR) was used to measure serum miR‐92a‐1 level. Relationship between miR‐92a‐1 and clinicopathological features of CRC cases was analysed via chi‐square test. Receiver operating characteristic (ROC) curve was plotted to estimate the diagnostic value of miR‐92a‐1 in CRC. Serum miR‐92a‐1 was significantly up‐regulated in CRC patients compared with healthy individuals (P < .001). Moreover, miR‐92a‐1 expression was correlated with TNM stage (P = .02), histological stage (P = .003), lymph node metastasis (P = .003) and distant metastasis (P < .001). ROC analysis showed that the area under the ROC curve (AUC) was 0.914, suggesting high diagnostic accuracy of miR‐92a‐1 in ROC. The optimal cut‐off value was 1.485, with a sensitivity of 81.8% and a specificity of 95.6%. MiR‐92a‐1 is increased in CRC patients and correlated with aggressive clinical characteristics. Serum miR‐92a‐1 may be a potential diagnostic biomarker for CRC.     

Bayesian and influence function-based empirical likelihoods for inference of sensitivity to the early diseased stage in diagnostic tests

In practice, a disease process might involve three ordinal diagnostic stages: the normal healthy stage, the early stage of the disease, and the stage of full development of the disease. Early detection is critical for some diseases since it often means an optimal time window for therapeutic treatments of the diseases. In this study, we propose a new influence function-based empirical likelihood method and Bayesian empirical likelihood methods to construct confidence/credible intervals for the sensitivity of a test to patients in the early diseased stage given a specificity and a sensitivity of the test to patients in the fully diseased stage. Numerical studies are performed to compare the finite sample performances of the proposed approaches with existing methods. The proposed methods are shown to outperform existing methods in terms of coverage probability. A real dataset from the Alzheimer's Disease Neuroimaging Initiative (ANDI) is used to illustrate the proposed methods.     

A modified sign test for comparing paired ROC curves

We develop a permutation test for assessing a difference in the areas under the curve (AUCs) in a paired setting where both modalities are given to each diseased and nondiseased subject. We propose that permutations be made between subjects specifically by shuffling the diseased/nondiseased labels of the subjects within each modality. As these permutations are made within modality, the permutation test is valid even if both modalities are measured on different scales. We show that our permutation test is a sign test for the symmetry of an underlying discrete distribution whose size remains valid under the assumption of equal AUCs. We demonstrate the operating characteristics of our test via simulation and show that our test is equal in power to a permutation test recently proposed by Bandos and others (2005).