Comparison of Diagnostic Accuracy of Biomarkers With Pooled Assessments

New biomarkers are frequently being developed in laboratory settings for the early diagnosis of diseases. However, the assay can be so expensive to assess in some cases that the evaluation of a large number of assays becomes unfeasible. Under this setting pooling biospecimens becomes an appealing alternative. In this paper, we present the methodology to allow for general pooling strategies and different data structures, which include balanced and unbalanced pooling cases. An estimate of the area under the ROC curve of a single biomarker with its asymptotic mean and variance is provided. Furthermore, we develop a test statistic for comparing the areas under the ROC curves of two biomarkers. The methods are illustrated with data from a study evaluating biomarkers for coronary heart disease.     

ROC曲线形状在生态位模型评价中的重要性——以美国白蛾为例

【目的】生态位模型在生物地理学、入侵生物学和保护生物学中具有广泛的应用,被越来越多地用于预测物种潜在分布和现实分布的研究中。本文以美国白蛾为例介绍pROC方案在生态位模型评价中的应用及其注意事项,以期对物种潜在分布预测进行合理的评价,促进生态位模型在我国的合理运用和发展。【方法】介绍ROC曲线和AUC值基本原理,总结其在生态位模型评价中的应用,从物种存在分布点和不存在分布点的可信度出发,分析AUC值用于模型评价的优点和不足,最后介绍局部受试者工作特征曲线的线下面积方案(pROC方案)来弥补传统AUC值的不足。【结果】AUC值虽独立于阈值,但因其综合灵敏度和特异度,而屏蔽这2个指标各自的特征,不能分别评估预测结果的灵敏度和特异度,同时对遗漏率和记账错率不能进行权衡,会误导使用者对模型的评价。与AUC值相比,ROC曲线的形状更具有价值,蕴含丰富的模型评价信息。【结论】模型评价需要将灵敏度和特异度区别对待,ROC曲线形状比AUC值在生态位模型评价中更为重要,pROC方案相对于传统AUC值具有优势,但容易对过度模拟做出不当判断。模型评价与作者研究目的密切相关:当以预测物种潜在分布为目的时(如入侵物种潜在分布、气候变化对物种分布的影响和谱系生物地理学),模型评价应当给予灵敏度(或者遗漏率)更多的权重;当以预测物种现实分布为目的时(如保护区界定和濒危物种引入),模型评价应当给予灵敏度和特异度同等的权重。     

ROC curve inference for best linear combination of two biomarkers subject to limits of detection

The receiver operating characteristic (ROC) curve is a tool commonly used to evaluate biomarker utility in clinical diagnosis of disease. Often, multiple biomarkers are developed to evaluate the discrimination for the same outcome. Levels of multiple biomarkers can be combined via best linear combination (BLC) such that their overall discriminatory ability is greater than any of them individually. Biomarker measurements frequently have undetectable levels below a detection limit sometimes denoted as limit of detection (LOD). Ignoring observations below the LOD or substituting some replacement value as a method of correction has been shown to lead to negatively biased estimates of the area under the ROC curve for some distributions of single biomarkers. In this paper, we develop asymptotically unbiased estimators, via the maximum likelihood technique, of the area under the ROC curve of BLC of two bivariate normally distributed biomarkers affected by LODs. We also propose confidence intervals for this area under curve. Point and confidence interval estimates are scrutinized by simulation study, recording bias and root mean square error and coverage probability, respectively. An example using polychlorinated biphenyl (PCB) levels to classify women with and without endometriosis illustrates the potential benefits of our methods.     

Robust Estimation of Area Under ROC Curve Using Auxiliary Variables in the Presence of Missing Biomarker Values

Summary In medical research, the receiver operating characteristic (ROC) curves can be used to evaluate the performance of biomarkers for diagnosing diseases or predicting the risk of developing a disease in the future. The area under the ROC curve (ROC AUC), as a summary measure of ROC curves, is widely utilized, especially when comparing multiple ROC curves. In observational studies, the estimation of the AUC is often complicated by the presence of missing biomarker values, which means that the existing estimators of the AUC are potentially biased. In this article, we develop robust statistical methods for estimating the ROC AUC and the proposed methods use information from auxiliary variables that are potentially predictive of the missingness of the biomarkers or the missing biomarker values. We are particularly interested in auxiliary variables that are predictive of the missing biomarker values. In the case of missing at random (MAR), that is, missingness of biomarker values only depends on the observed data, our estimators have the attractive feature of being consistent if one correctly specifies, conditional on auxiliary variables and disease status, either the model for the probabilities of being missing or the model for the biomarker values. In the case of missing not at random (MNAR), that is, missingness may depend on the unobserved biomarker values, we propose a sensitivity analysis to assess the impact of MNAR on the estimation of the ROC AUC. The asymptotic properties of the proposed estimators are studied and their finite‐sample behaviors are evaluated in simulation studies. The methods are further illustrated using data from a study of maternal depression during pregnancy.     

癌胚抗原蛋白芯片的研发及其在肝癌检测中的评价

目的:研究建立一种简便、快速、特异性高、低成本的检测血清中癌胚抗原(CEA)浓度的蛋白芯片,并通过检测肝细胞肝癌(HCC)对其进行评价。方法:采用双抗体夹心法,制备能够形成捕获抗体-抗原-检测抗体的"三明治"结构的蛋白芯片检测血清中CEA浓度。通过用该蛋白芯片检测50例CEA阳性HCC患者血清和56例健康人血清,对其进行盲法验证。结果:以CEA5 ng/m L为阳性判定标准,得出CEA蛋白芯片的灵敏度为92%(46/50),特异度为100%(56/56)。受试者工作特征(ROC)曲线分析显示该蛋白芯片检测出血清CEA的ROC曲线下面积(AUC)为0.960,与0.5相比差异有统计学意义(P0.001),其判定CEA阳性的准确性较高。结论:成功建立检测血清中CEA浓度的蛋白芯片,为下一步研发多种标志物联合检测HCC的蛋白芯片提供候选血清标志物。     

Youden Index and optimal cut-point estimated from observations affected by a lower limit of detection

The receiver operating characteristic (ROC) curve is used to evaluate a biomarker's ability for classifying disease status. The Youden Index (J), the maximum potential effectiveness of a biomarker, is a common summary measure of the ROC curve. In biomarker development, levels may be unquantifiable below a limit of detection (LOD) and missing from the overall dataset. Disregarding these observations may negatively bias the ROC curve and thus J. Several correction methods have been suggested for mean estimation and testing; however, little has been written about the ROC curve or its summary measures. We adapt non-parametric (empirical) and semi-parametric (ROC-GLM [generalized linear model]) methods and propose parametric methods (maximum likelihood (ML)) to estimate J and the optimal cut-point (c *) for a biomarker affected by a LOD. We develop unbiased estimators of J and c * via ML for normally and gamma distributed biomarkers. Alpha level confidence intervals are proposed using delta and bootstrap methods for the ML, semi-parametric, and non-parametric approaches respectively. Simulation studies are conducted over a range of distributional scenarios and sample sizes evaluating estimators' bias, root-mean square error, and coverage probability; the average bias was less than one percent for ML and GLM methods across scenarios and decreases with increased sample size. An example using polychlorinated biphenyl levels to classify women with and without endometriosis illustrates the potential benefits of these methods. We address the limitations and usefulness of each method in order to give researchers guidance in constructing appropriate estimates of biomarkers' true discriminating capabilities.     

Application of Artificial Intelligence/Machine Vision & Learning for the Development of a Live Single-cell Phenotypic Biomarker Test to Predict Prostate Cancer Tumor Aggressiveness

To assess the usefulness and applications of machine vision (MV) and machine learning (ML) techniques that have been used to develop a single cell-based phenotypic (live and fixed biomarkers) platform that correlates with tumor biological aggressiveness and risk stratification, 100 fresh prostate samples were acquired, and areas of prostate cancer were determined by post-surgery pathology reports logged by an independent pathologist. The prostate samples were dissociated into single-cell suspensions in the presence of an extracellular matrix formulation. These samples were analyzed via live-cell microscopy. Dynamic and fixed phenotypic biomarkers per cell were quantified using objective MV software and ML algorithms. The predictive nature of the ML algorithms was developed in two stages. First, random forest (RF) algorithms were developed using 70% of the samples. The developed algorithms were then tested for their predictive performance using the blinded test dataset that contained 30% of the samples in the second stage. Based on the ROC (receiver operating characteristic) curve analysis, thresholds were set to maximize both sensitivity and specificity. We determined the sensitivity and specificity of the assay by comparing the algorithm-generated predictions with adverse pathologic features in the radical prostatectomy (RP) specimens. Using MV and ML algorithms, the biomarkers predictive of adverse pathology at RP were ranked and a prostate cancer patient risk stratification test was developed that distinguishes patients based on surgical adverse pathology features. The ability to identify and track large numbers of individual cells over the length of the microscopy experimental monitoring cycles, in an automated way, created a large biomarker dataset of primary biomarkers. This biomarker dataset was then interrogated with ML algorithms used to correlate with post-surgical adverse pathology findings. Algorithms were generated that predicted adverse pathology with >0.85 sensitivity and specificity and an AUC (area under the curve) of >0.85. Phenotypic biomarkers provide cellular and molecular details that are informative for predicting post-surgical adverse pathologies when considering tumor biopsy samples. Artificial intelligence ML-based approaches for cancer risk stratification are emerging as important and powerful tools to compliment current measures of risk stratification. These techniques have capabilities to address tumor heterogeneity and the molecular complexity of prostate cancer. Specifically, the phenotypic test is a novel example of leveraging biomarkers and advances in MV and ML for developing a powerful prognostic and risk-stratification tool for prostate cancer patients.     

Exploration of Analysis Methods for Diagnostic Imaging Tests: Problems with ROC AUC and Confidence Scores in CT Colonography

Background

Different methods of evaluating diagnostic performance when comparing diagnostic tests may lead to different results. We compared two such approaches, sensitivity and specificity with area under the Receiver Operating Characteristic Curve (ROC AUC) for the evaluation of CT colonography for the detection of polyps, either with or without computer assisted detection.

Methods

In a multireader multicase study of 10 readers and 107 cases we compared sensitivity and specificity, using radiological reporting of the presence or absence of polyps, to ROC AUC calculated from confidence scores concerning the presence of polyps. Both methods were assessed against a reference standard. Here we focus on five readers, selected to illustrate issues in design and analysis. We compared diagnostic measures within readers, showing that differences in results are due to statistical methods.

Results

Reader performance varied widely depending on whether sensitivity and specificity or ROC AUC was used. There were problems using confidence scores; in assigning scores to all cases; in use of zero scores when no polyps were identified; the bimodal non-normal distribution of scores; fitting ROC curves due to extrapolation beyond the study data; and the undue influence of a few false positive results. Variation due to use of different ROC methods exceeded differences between test results for ROC AUC.

Conclusions

The confidence scores recorded in our study violated many assumptions of ROC AUC methods, rendering these methods inappropriate. The problems we identified will apply to other detection studies using confidence scores. We found sensitivity and specificity were a more reliable and clinically appropriate method to compare diagnostic tests.     

Insights into the area under the receiver operating characteristic curve (AUC) as a discrimination measure in species distribution modelling

Aim The area under the receiver operating characteristic (ROC) curve (AUC) is a widely used statistic for assessing the discriminatory capacity of species distribution models. Here, I used simulated data to examine the interdependence of the AUC and classical discrimination measures (sensitivity and specificity) derived for the application of a threshold. I shall further exemplify with simulated data the implications of using the AUC to evaluate potential versus realized distribution models. Innovation After applying the threshold that makes sensitivity and specificity equal, a strong relationship between the AUC and these two measures was found. This result is corroborated with real data. On the other hand, the AUC penalizes the models that estimate potential distributions (the regions where the species could survive and reproduce due to the existence of suitable environmental conditions), and favours those that estimate realized distributions (the regions where the species actually lives). Main conclusions Firstly, the independence of the AUC from the threshold selection may be irrelevant in practice. This result also emphasizes the fact that the AUC assumes nothing about the relative costs of errors of omission and commission. However, in most real situations this premise may not be optimal. Measures derived from a contingency table for different cost ratio scenarios, together with the ROC curve, may be more informative than reporting just a single AUC value. Secondly, the AUC is only truly informative when there are true instances of absence available and the objective is the estimation of the realized distribution. When the potential distribution is the goal of the research, the AUC is not an appropriate performance measure because the weight of commission errors is much lower than that of omission errors.     

Maximum likelihood ratio tests for comparing the discriminatory ability of biomarkers subject to limit of detection

Summary .   In this article, we consider comparing the areas under correlated receiver operating characteristic (ROC) curves of diagnostic biomarkers whose measurements are subject to a limit of detection (LOD), a source of measurement error from instruments' sensitivity in epidemiological studies. We propose and examine the likelihood ratio tests with operating characteristics that are easily obtained by classical maximum likelihood methodology.     

Translational biomarker discovery in clinical metabolomics: an introductory tutorial

Metabolomics is increasingly being applied towards the identification of biomarkers for disease diagnosis, prognosis and risk prediction. Unfortunately among the many published metabolomic studies focusing on biomarker discovery, there is very little consistency and relatively little rigor in how researchers select, assess or report their candidate biomarkers. In particular, few studies report any measure of sensitivity, specificity, or provide receiver operator characteristic (ROC) curves with associated confidence intervals. Even fewer studies explicitly describe or release the biomarker model used to generate their ROC curves. This is surprising given that for biomarker studies in most other biomedical fields, ROC curve analysis is generally considered the standard method for performance assessment. Because the ultimate goal of biomarker discovery is the translation of those biomarkers to clinical practice, it is clear that the metabolomics community needs to start “speaking the same language” in terms of biomarker analysis and reporting-especially if it wants to see metabolite markers being routinely used in the clinic. In this tutorial, we will first introduce the concept of ROC curves and describe their use in single biomarker analysis for clinical chemistry. This includes the construction of ROC curves, understanding the meaning of area under ROC curves (AUC) and partial AUC, as well as the calculation of confidence intervals. The second part of the tutorial focuses on biomarker analyses within the context of metabolomics. This section describes different statistical and machine learning strategies that can be used to create multi-metabolite biomarker models and explains how these models can be assessed using ROC curves. In the third part of the tutorial we discuss common issues and potential pitfalls associated with different analysis methods and provide readers with a list of nine recommendations for biomarker analysis and reporting. To help readers test, visualize and explore the concepts presented in this tutorial, we also introduce a web-based tool called ROCCET (ROC Curve Explorer & Tester, http://www.roccet.ca). ROCCET was originally developed as a teaching aid but it can also serve as a training and testing resource to assist metabolomics researchers build biomarker models and conduct a range of common ROC curve analyses for biomarker studies.     

Combining biomarkers to detect disease with application to prostate cancer

In early detection of disease, combinations of biomarkers promise improved discrimination over diagnostic tests based on single markers. An example of this is in prostate cancer screening, where additional markers have been sought to improve the specificity of the conventional Prostate-Specific Antigen (PSA) test. A marker of particular interest is the percent free PSA. Studies evaluating the benefits of percent free PSA reflect the need for a methodological approach that is statistically valid and useful in the clinical setting. This article presents methods that address this need. We focus on and-or combinations of biomarker results that we call logic rules and present novel definitions for the ROC curve and the area under the curve (AUC) that are applicable to this class of combination tests. Our estimates of the ROC and AUC are amenable to statistical inference including comparisons of tests and regression analysis. The methods are applied to data on free and total PSA levels among prostate cancer cases and matched controls enrolled in the Physicians' Health Study.     

A multiplex platform for the identification of ovarian cancer biomarkers

Background

Currently, there are no FDA approved screening tools for detecting early stage ovarian cancer in the general population. Development of a biomarker-based assay for early detection would significantly improve the survival of ovarian cancer patients.

Methods

We used a multiplex approach to identify protein biomarkers for detecting early stage ovarian cancer. This new technology (Proseek^® Multiplex Oncology Plates) can simultaneously measure the expression of 92 proteins in serum based on a proximity extension assay. We analyzed serum samples from 81 women representing healthy, benign pathology, early, and advanced stage serous ovarian cancer patients.

Results

Principle component analysis and unsupervised hierarchical clustering separated patients into cancer versus non-cancer subgroups. Data from the Proseek^® plate for CA125 levels exhibited a strong correlation with current clinical assays for CA125 (correlation coefficient of 0.89, 95% CI 0.83, 0.93). CA125 and HE4 were present at very low levels in healthy controls and benign cases, while higher levels were found in early stage cases, with highest levels found in the advanced stage cases. Overall, significant trends were observed for 38 of the 92 proteins (p < 0.001), many of which are novel candidate serum biomarkers for ovarian cancer. The area under the ROC curve (AUC) for CA125 was 0.98 and the AUC for HE4 was 0.85 when comparing early stage ovarian cancer versus healthy controls. In total, 23 proteins had an estimated AUC of 0.7 or greater. Using a naïve Bayes classifier that combined 12 proteins, we improved the sensitivity corresponding to 95% specificity from 93 to 95% when compared to CA125 alone. Although small, a 2% increase would have a significant effect on the number of women correctly identified when screening a large population.

Conclusions

These data demonstrate that the Proseek^® technology can replicate the results established by conventional clinical assays for known biomarkers, identify new candidate biomarkers, and improve the sensitivity and specificity of CA125 alone. Additional studies using a larger cohort of patients will allow for validation of these biomarkers and lead to the development of a screening tool for detecting early stage ovarian cancer in the general population.

     

Threshold-dependence as a desirable attribute for discrimination assessment: implications for the evaluation of species distribution models

Species distribution modelling has become a common approach in ecology in the last decades. As in any modelling exercise, evaluation of the predicted suitability surfaces is a key process, and the area under the receiver operating characteristic (ROC) curve (AUC) has become the most popular statistic for this purpose. A close covariation between the AUC and threshold-dependent discrimination measures (sensitivity Se and specificity Sp) raises into question the advantage of the threshold-independence of the AUC. In this study, the relationship between the AUC and several threshold-dependent discrimination measures is characterized in detail, and the sensitivity of the pattern to variations in the shape of the ROC curve is assessed. Hypothetical suitability values, coming from normal and skew-normal distributions, were simulated for both instances of presence and absence. The flexibility of the skew-normal distribution allowed for the simulation of a wide range of ROC curve configurations. The relationship between the AUC and threshold-dependent measures was graphically assessed; independently of the ROC curve shape, a nonlinear asymptotic relationship between the AUC and Se (and Sp) was obtained after applying the threshold that makes Se = Sp. A nonlinear asymptotic relationship between the AUC and the Youden index was also reported. These results imply that the AUC does not appropriately measure changes in the discrimination of models, and it is especially incapable of distinguishing between models with high discrimination capacity. Se or Sp derived from the application of the threshold that makes them equal is a preferred measure of discrimination power. Together with the rate of false positives and negatives, and with the prevalence of the species, these statistics provide more information about the discrimination capacity of the models than the AUC.     

Plasma miR-601 and miR-760 Are Novel Biomarkers for the Early Detection of Colorectal Cancer

Background

Colorectal cancer (CRC) is a major cause of death worldwide. Sensitive, non-invasive diagnostic screen methods are urgently needed to improve its survival rates. Stable circulating microRNA offers unique opportunities for the early diagnosis of several diseases, including cancers. Our aim has been to find new plasma miRNAs that can be used as biomarkers for the detection of CRC.

Methodology/Principal Findings

According to the results of miRNA profiling performed on pooling plasma samples form 10 CRC patients or 10 healthy controls, a panel of miRNAs (hsa-miR-10a, -19a, -22*, -24, -92a, 125a-5p, -141, -150, -188-3p, -192, -210, -221, -224*, -376a, -425*, -495, -572, -601, -720, -760 and hsa-let-7a, -7e) were deregulated in CRC plasma with fold changes >5. After large scale validation by qRT-PCR performed on another 191 independent individuals (90 CRC, 43 advanced adenoma and 58 healthy participants), we found that the levels of plasma miR-601 and miR-760 were significantly decreased in colorectal neoplasia (carcinomas and advanced adenomas) compared with healthy controls. ROC curve analysis showed that plasma miR-601 and miR-760 were of significant diagnostic value for advanced neoplasia. These two miRNAs together yield an AUC of 0.792 with 83.3% sensitivity and 69.1% specificity for separating CRC from normal controls, and yield an AUC of 0.683 with 72.1% sensitivity and 62.1% specificity in discriminating advanced adenomas from normal controls.

Conclusions/Significance

Plasma miR-601 and miR-760 can potentially serve as promising non-invasive biomarkers for the early detection of CRC.     

On the use of partial area under the ROC curve for comparison of two diagnostic tests

Evaluation of diagnostic performance is typically based on the receiver operating characteristic (ROC) curve and the area under the curve (AUC) as its summary index. The partial area under the curve (pAUC) is an alternative index focusing on the range of practical/clinical relevance. One of the problems preventing more frequent use of the pAUC is the perceived loss of efficiency in cases of noncrossing ROC curves. In this paper, we investigated statistical properties of comparisons of two correlated pAUCs. We demonstrated that outside of the classic model there are practically reasonable ROC types for which comparisons of noncrossing concave curves would be more powerful when based on a part of the curve rather than the entire curve. We argue that this phenomenon stems in part from the exclusion of noninformative parts of the ROC curves that resemble straight‐lines. We conducted extensive simulation studies in families of binormal, straight‐line, and bigamma ROC curves. We demonstrated that comparison of pAUCs is statistically more powerful than comparison of full AUCs when ROC curves are close to a “straight line”. For less flat binormal ROC curves an increase in the integration range often leads to a disproportional increase in pAUCs’ difference, thereby contributing to an increase in statistical power. Thus, efficiency of differences in pAUCs of noncrossing ROC curves depends on the shape of the curves, and for families of ROC curves that are nearly straight‐line shaped, such as bigamma ROC curves, there are multiple practical scenarios in which comparisons of pAUCs are preferable.     

The analysis of placement values for evaluating discriminatory measures

The idea of using measurements such as biomarkers, clinical data, or molecular biology assays for classification and prediction is popular in modern medicine. The scientific evaluation of such measures includes assessing the accuracy with which they predict the outcome of interest. Receiver operating characteristic curves are commonly used for evaluating the accuracy of diagnostic tests. They can be applied more broadly, indeed to any problem involving classification to two states or populations (D= 0 or 1). We show that the ROC curve can be interpreted as a cumulative distribution function for the discriminatory measure Y in the affected population (D= 1) after Y has been standardized to the distribution in the reference population (D= 0). The standardized values are called placement values. If the placement values have a uniform(0, 1) distribution, then Y is not discriminatory, because its distribution in the affected population is the same as that in the reference population. The degree to which the distribution of the standardized measure differs from uniform(0, 1) is a natural way to characterize the discriminatory capacity of Y and provides a nontraditional interpretation for the ROC curve. Statistical methods for making inference about distribution functions therefore motivate new approaches to making inference about ROC curves. We demonstrate this by considering the ROC-GLM regression model and observing that it is equivalent to a regression model for the distribution of placement values. The likelihood of the placement values provides a new approach to ROC parameter estimation that appears to be more efficient than previously proposed methods. The method is applied to evaluate a pulmonary function measure in cystic fibrosis patients as a predictor of future occurrence of severe acute pulmonary infection requiring hospitalization. Finally, we note the relationship between regression models for the mean placement value and recently proposed models for the area under the ROC curve which is the classic summary index of discrimination.     

Diagnosis of pancreatic ductal adenocarcinoma and chronic pancreatitis by measurement of microRNA abundance in blood and tissue

A solid process for diagnosis could have a substantial impact on the successful treatment of pancreatic cancer, for which currently mortality is nearly identical to incidence. Variations in the abundance of all microRNA molecules from peripheral blood cells and pancreas tissues were analyzed on microarrays and in part validated by real-time PCR assays. In total, 245 samples from two clinical centers were studied that were obtained from patients with pancreatic ductal adenocarcinoma or chronic pancreatitis and from healthy donors. Utilizing the minimally invasive blood test, receiver operating characteristic (ROC) curves and the corresponding area under the curve (AUC) analysis demonstrated very high sensitivity and specificity of a distinction between healthy people and patients with either cancer or chronic pancreatitis; respective AUC values of 0.973 and 0.950 were obtained. Confirmative and partly even more discriminative diagnosis could be performed on tissue samples with AUC values of 1.0 and 0.937, respectively. In addition, discrimination between cancer and chronic pancreatitis was achieved (AUC = 0.875). Also, several miRNAs were identified that exhibited abundance variations in both tissue and blood samples. The results could have an immediate diagnostic value for the evaluation of tumor reoccurrence in patients, who have undergone curative surgical resection, and for people with a familial risk of pancreatic cancer.