ROC曲线形状在生态位模型评价中的重要性——以美国白蛾为例

【目的】生态位模型在生物地理学、入侵生物学和保护生物学中具有广泛的应用,被越来越多地用于预测物种潜在分布和现实分布的研究中。本文以美国白蛾为例介绍pROC方案在生态位模型评价中的应用及其注意事项,以期对物种潜在分布预测进行合理的评价,促进生态位模型在我国的合理运用和发展。【方法】介绍ROC曲线和AUC值基本原理,总结其在生态位模型评价中的应用,从物种存在分布点和不存在分布点的可信度出发,分析AUC值用于模型评价的优点和不足,最后介绍局部受试者工作特征曲线的线下面积方案(pROC方案)来弥补传统AUC值的不足。【结果】AUC值虽独立于阈值,但因其综合灵敏度和特异度,而屏蔽这2个指标各自的特征,不能分别评估预测结果的灵敏度和特异度,同时对遗漏率和记账错率不能进行权衡,会误导使用者对模型的评价。与AUC值相比,ROC曲线的形状更具有价值,蕴含丰富的模型评价信息。【结论】模型评价需要将灵敏度和特异度区别对待,ROC曲线形状比AUC值在生态位模型评价中更为重要,pROC方案相对于传统AUC值具有优势,但容易对过度模拟做出不当判断。模型评价与作者研究目的密切相关:当以预测物种潜在分布为目的时(如入侵物种潜在分布、气候变化对物种分布的影响和谱系生物地理学),模型评价应当给予灵敏度(或者遗漏率)更多的权重;当以预测物种现实分布为目的时(如保护区界定和濒危物种引入),模型评价应当给予灵敏度和特异度同等的权重。     

Robust Estimation of Area Under ROC Curve Using Auxiliary Variables in the Presence of Missing Biomarker Values

Summary In medical research, the receiver operating characteristic (ROC) curves can be used to evaluate the performance of biomarkers for diagnosing diseases or predicting the risk of developing a disease in the future. The area under the ROC curve (ROC AUC), as a summary measure of ROC curves, is widely utilized, especially when comparing multiple ROC curves. In observational studies, the estimation of the AUC is often complicated by the presence of missing biomarker values, which means that the existing estimators of the AUC are potentially biased. In this article, we develop robust statistical methods for estimating the ROC AUC and the proposed methods use information from auxiliary variables that are potentially predictive of the missingness of the biomarkers or the missing biomarker values. We are particularly interested in auxiliary variables that are predictive of the missing biomarker values. In the case of missing at random (MAR), that is, missingness of biomarker values only depends on the observed data, our estimators have the attractive feature of being consistent if one correctly specifies, conditional on auxiliary variables and disease status, either the model for the probabilities of being missing or the model for the biomarker values. In the case of missing not at random (MNAR), that is, missingness may depend on the unobserved biomarker values, we propose a sensitivity analysis to assess the impact of MNAR on the estimation of the ROC AUC. The asymptotic properties of the proposed estimators are studied and their finite‐sample behaviors are evaluated in simulation studies. The methods are further illustrated using data from a study of maternal depression during pregnancy.     

A model free approach to combining biomarkers

For most diseases, single biomarkers do not have adequate sensitivity or specificity for practical purposes. We present an approach to combine several biomarkers into a composite marker score without assuming a model for the distribution of the predictors. Using sufficient dimension reduction techniques, we replace the original markers with a lower-dimensional version, obtained through linear transformations of markers that contain sufficient information for regression of the predictors on the outcome. We combine the linear transformations using their asymptotic properties into a scalar diagnostic score via the likelihood ratio statistic. The performance of this score is assessed by the area under the receiver-operator characteristics curve (ROC), a popular summary measure of the discriminatory ability of a single continuous diagnostic marker for binary disease outcomes. An asymptotic chi-squared test for assessing individual biomarker contribution to the diagnostic score is also derived.     

Semi-parametric ROC regression analysis with placement values

Advances in technology provide new diagnostic tests for early detection of disease. Frequently, these tests have continuous outcomes. One popular method to summarize the accuracy of such a test is the Receiver Operating Characteristic (ROC) curve. Methods for estimating ROC curves have long been available. To examine covariate effects, Pepe (1997, 2000) and Alonzo and Pepe (2002) proposed distribution-free approaches based on a parametric regression model for the ROC curve. Cai and Pepe (2002) extended the parametric ROC regression model by allowing an arbitrary non-parametric baseline function. In this paper, while we follow the same semi-parametric setting as in that paper, we highlight a new estimator that offers several improvements over the earlier work: superior efficiency, the ability to estimate the covariate effects without estimating the non-parametric baseline function and easy implementation with standard software. The methodology is applied to a case control dataset where we evaluate the accuracy of the prostate-specific antigen as a biomarker for early detection of prostate cancer. Simulation studies suggest that the new estimator under the semi-parametric model, while always being more robust, has efficiency that is comparable to or better than the Alonzo and Pepe (2002) estimator from the parametric model.     

梗阻性黄疸MRCP 的循证和临床研究

目的：通过meta、ROC分析以及按病变部位、性质进行的亚组分析分析对目前诊断梗阻性黄疸的非侵入性影像诊断方法（US,Cr和MRCP）进行对比研究。方法：1、采用medline检索。纳入标准为：（a）US、CT和MRCP诊断梗阻性黄疸性疾病的文献（b）病理检查、术中所见或临床、实验室检查结果作为诊断金标准。（c）能够直接或间接获得每个影像方法的真、假阳性数,真、假阴性数。提取数据、文献质量评估通过kappa分析进行一致性检验。统计分析采用漏斗图、SROC分析方法以及协变量分析。2、疑胆胰系疾患接受MRCP检查患者105例,其中同时做US检查者65例。另有同期Cr资料59例,其中同时做US检查者31例。盲法与金标准对比,计算出各诊断方法的真阳性率和假阳性率,ROC分析其诊断效能。同时按病变部位、性质分别计算MR-CP、US及Cr的敏感度、特异度和似然比等指标进行比较分析。结果：1、漏斗图US相关文献分布形状略不规则,CT、MRCP相关文献分布形状类似漏斗形。SROC曲线图MRCP线最靠近左上角,诊断效能高于US和CT、MRCP的Q^＊值（0．9256）高于US（0.8765）和CT（0.8606）。三者间经检验无显著性差别,MRCP和cT问检验Z=0、33,双侧P〉0、25。协变量分析未见对诊断效能有显著性影响因素。2、ROC分析显示,MRCP的曲线最靠近左上角,US次之,Cr在最下面,三者的曲线下面积（Az）分别为0．985,0．981．0、901,均大于0、9,MRCP与Cr间离均差（Z）为0．75,双侧P〉0、25。MRCP、US和Cr诊断胆胰系恶性占位、结石的敏感度分别为100％、83％、82％;92％、71％、76％。经检验,MRCP与US和CT间有显著性差异,P〈0．05。结论：经meta、ROC分析,认为MRCP在诊断梗阻性黄疸疾病中具有优势,诊断效能高于US和Cr。     

Skew-normal random-effects model for meta-analysis of diagnostic test accuracy (DTA) studies

Hierarchical models are recommended for meta-analyzing diagnostic test accuracy (DTA) studies. The bivariate random-effects model is currently widely used to synthesize a pair of test sensitivity and specificity using logit transformation across studies. This model assumes a bivariate normal distribution for the random-effects. However, this assumption is restrictive and can be violated. When the assumption fails, inferences could be misleading. In this paper, we extended the current bivariate random-effects model by assuming a flexible bivariate skew-normal distribution for the random-effects in order to robustly model logit sensitivities and logit specificities. The marginal distribution of the proposed model is analytically derived so that parameter estimation can be performed using standard likelihood methods. The method of weighted-average is adopted to estimate the overall logit-transformed sensitivity and specificity. An extensive simulation study is carried out to investigate the performance of the proposed model compared to other standard models. Overall, the proposed model performs better in terms of confidence interval width of the average logit-transformed sensitivity and specificity compared to the standard bivariate linear mixed model and bivariate generalized linear mixed model. Simulations have also shown that the proposed model performed better than the well-established bivariate linear mixed model in terms of bias and comparable with regards to the root mean squared error (RMSE) of the between-study (co)variances. The proposed method is also illustrated using a published meta-analysis data.     

Translational biomarker discovery in clinical metabolomics: an introductory tutorial

Metabolomics is increasingly being applied towards the identification of biomarkers for disease diagnosis, prognosis and risk prediction. Unfortunately among the many published metabolomic studies focusing on biomarker discovery, there is very little consistency and relatively little rigor in how researchers select, assess or report their candidate biomarkers. In particular, few studies report any measure of sensitivity, specificity, or provide receiver operator characteristic (ROC) curves with associated confidence intervals. Even fewer studies explicitly describe or release the biomarker model used to generate their ROC curves. This is surprising given that for biomarker studies in most other biomedical fields, ROC curve analysis is generally considered the standard method for performance assessment. Because the ultimate goal of biomarker discovery is the translation of those biomarkers to clinical practice, it is clear that the metabolomics community needs to start “speaking the same language” in terms of biomarker analysis and reporting-especially if it wants to see metabolite markers being routinely used in the clinic. In this tutorial, we will first introduce the concept of ROC curves and describe their use in single biomarker analysis for clinical chemistry. This includes the construction of ROC curves, understanding the meaning of area under ROC curves (AUC) and partial AUC, as well as the calculation of confidence intervals. The second part of the tutorial focuses on biomarker analyses within the context of metabolomics. This section describes different statistical and machine learning strategies that can be used to create multi-metabolite biomarker models and explains how these models can be assessed using ROC curves. In the third part of the tutorial we discuss common issues and potential pitfalls associated with different analysis methods and provide readers with a list of nine recommendations for biomarker analysis and reporting. To help readers test, visualize and explore the concepts presented in this tutorial, we also introduce a web-based tool called ROCCET (ROC Curve Explorer & Tester, http://www.roccet.ca). ROCCET was originally developed as a teaching aid but it can also serve as a training and testing resource to assist metabolomics researchers build biomarker models and conduct a range of common ROC curve analyses for biomarker studies.     

Regularized ROC method for disease classification and biomarker selection with microarray data

MOTIVATION: An important application of microarrays is to discover genomic biomarkers, among tens of thousands of genes assayed, for disease classification. Thus there is a need for developing statistical methods that can efficiently use such high-throughput genomic data, select biomarkers with discriminant power and construct classification rules. The ROC (receiver operator characteristic) technique has been widely used in disease classification with low-dimensional biomarkers because (1) it does not assume a parametric form of the class probability as required for example in the logistic regression method; (2) it accommodates case-control designs and (3) it allows treating false positives and false negatives differently. However, due to computational difficulties, the ROC-based classification has not been used with microarray data. Moreover, the standard ROC technique does not incorporate built-in biomarker selection. RESULTS: We propose a novel method for biomarker selection and classification using the ROC technique for microarray data. The proposed method uses a sigmoid approximation to the area under the ROC curve as the objective function for classification and the threshold gradient descent regularization method for estimation and biomarker selection. Tuning parameter selection based on the V-fold cross validation and predictive performance evaluation are also investigated. The proposed approach is demonstrated with a simulation study, the Colon data and the Estrogen data. The proposed approach yields parsimonious models with excellent classification performance.     

血浆HOTAIR在冠状动脉粥样硬化性心脏病的表达及诊断价值

目的:检测血浆HOX转录反义RNA(HOTAIR)表达水平,并初步探讨血浆HOTAIR作为冠状动脉粥样硬化性心脏病(CAD)标志物的诊断价值。方法:CAD患者和非CAD患者分别作为本研究的疾病组和对照组,血标本来自重庆大坪医院心内科介入室(2013年1月-9月)。运用实时荧光定量聚合酶链反应(q RT-PCR)分别检测疾病组和对照组各100例血浆HOTAIR水平,同时对上述血浆HOTAIR的表达值绘制受试者工作特征(ROC)曲线,并计算ROC曲线下面积(AUC)。结果:与对照组(0.25±0.10)相比,HOTAIR在CAD患者血浆(0.80±0.30)中表达显著升高(P0.05);ROC曲线分析表明,AUC为0.776,Cut-off值为0.55时,诊断CAD的灵敏度为75%,特异度为80%。结论:冠状动脉粥样硬化性心脏病患者血浆HOTAIR水平升高,血浆HOTAIR可作为诊断冠状动脉粥样硬化性心脏病的独立生物标志物。     

Insights into the area under the receiver operating characteristic curve (AUC) as a discrimination measure in species distribution modelling

Aim The area under the receiver operating characteristic (ROC) curve (AUC) is a widely used statistic for assessing the discriminatory capacity of species distribution models. Here, I used simulated data to examine the interdependence of the AUC and classical discrimination measures (sensitivity and specificity) derived for the application of a threshold. I shall further exemplify with simulated data the implications of using the AUC to evaluate potential versus realized distribution models. Innovation After applying the threshold that makes sensitivity and specificity equal, a strong relationship between the AUC and these two measures was found. This result is corroborated with real data. On the other hand, the AUC penalizes the models that estimate potential distributions (the regions where the species could survive and reproduce due to the existence of suitable environmental conditions), and favours those that estimate realized distributions (the regions where the species actually lives). Main conclusions Firstly, the independence of the AUC from the threshold selection may be irrelevant in practice. This result also emphasizes the fact that the AUC assumes nothing about the relative costs of errors of omission and commission. However, in most real situations this premise may not be optimal. Measures derived from a contingency table for different cost ratio scenarios, together with the ROC curve, may be more informative than reporting just a single AUC value. Secondly, the AUC is only truly informative when there are true instances of absence available and the objective is the estimation of the realized distribution. When the potential distribution is the goal of the research, the AUC is not an appropriate performance measure because the weight of commission errors is much lower than that of omission errors.     

Extending induced ROC methodology to the functional context

The receiver operating characteristic (ROC) curve is the most widely used measure for evaluating the discriminatory performance of a continuous marker. Often, covariate information is also available and several regression methods have been proposed to incorporate covariate information in the ROC framework. Until now, these methods are only developed for the case where the covariate is univariate or multivariate. We extend ROC regression methodology for the case where the covariate is functional rather than univariate or multivariate. To this end, semiparametric- and nonparametric-induced ROC regression estimators are proposed. A simulation study is performed to assess the performance of the proposed estimators. The methods are applied to and motivated by a metabolic syndrome study in Galicia (NW Spain).     

Assessing the predictive power of newly added biomarkers

As medical research and technology advance, there are always new biomarkers found and predictive models proposed for improving the diagnostic performance of diseases. Therefore, in addition to the existing biomarkers and predictive models, how to assess new biomarkers becomes an important research problem. Many classification performance measures, which are usually based on the performance on the whole cut‐off values, were applied directly to this type of problems. However, in a medical diagnosis, some cut‐off points are more important, such as those points within the range of high specificity. Thus, as the partial area under the ROC curve to the area under ROC curve, we study the partial integrated discriminant improvement (pIDI) for evaluating the predictive ability of a newly added marker at a prespecified range of cut‐offs. Theoretical property of estimate of the proposed measure is reported. The performance of this new measure is then compared with that of the partial area under an ROC curve. The numerical results use synthesized are presented, and a liver cancer dataset is used for demonstration purposes.     

Biomarker evaluation and comparison using the controls as a reference population

The classification accuracy of a continuous marker is typically evaluated with the receiver operating characteristic (ROC) curve. In this paper, we study an alternative conceptual framework, the "percentile value." In this framework, the controls only provide a reference distribution to standardize the marker. The analysis proceeds by analyzing the standardized marker in cases. The approach is shown to be equivalent to ROC analysis. Advantages are that it provides a framework familiar to a broad spectrum of biostatisticians and it opens up avenues for new statistical techniques in biomarker evaluation. We develop several new procedures based on this framework for comparing biomarkers and biomarker performance in different populations. We develop methods that adjust such comparisons for covariates. The methods are illustrated on data from 2 cancer biomarker studies.     

Smoothed empirical likelihood inference for ROC curve in the presence of missing biomarker values

This paper considers statistical inference for the receiver operating characteristic (ROC) curve in the presence of missing biomarker values by utilizing estimating equations (EEs) together with smoothed empirical likelihood (SEL). Three approaches are developed to estimate ROC curve and construct its SEL-based confidence intervals based on the kernel-assisted EE imputation, multiple imputation, and hybrid imputation combining the inverse probability weighted imputation and multiple imputation. Under some regularity conditions, we show asymptotic properties of the proposed maximum SEL estimators for ROC curve. Simulation studies are conducted to investigate the performance of the proposed SEL approaches. An example is illustrated by the proposed methodologies. Empirical results show that the hybrid imputation method behaves better than the kernel-assisted and multiple imputation methods, and the proposed three SEL methods outperform existing nonparametric method.     

Bivariate random‐effects meta‐analysis models for diagnostic test accuracy studies using arcsine‐based transformations

Diagnostic or screening tests are widely used in medical fields to classify patients according to their disease status. Several statistical models for meta‐analysis of diagnostic test accuracy studies have been developed to synthesize test sensitivity and specificity of a diagnostic test of interest. Because of the correlation between test sensitivity and specificity, modeling the two measures using a bivariate model is recommended. In this paper, we extend the current standard bivariate linear mixed model (LMM) by proposing two variance‐stabilizing transformations: the arcsine square root and the Freeman–Tukey double arcsine transformation. We compared the performance of the proposed methods with the standard method through simulations using several performance measures. The simulation results showed that our proposed methods performed better than the standard LMM in terms of bias, root mean square error, and coverage probability in most of the scenarios, even when data were generated assuming the standard LMM. We also illustrated the methods using two real data sets.     

Combining diagnostic test results to increase accuracy

When multiple diagnostic tests are performed on an individual or multiple disease markers are available it may be possible to combine the information to diagnose disease. We consider how to choose linear combinations of markers in order to optimize diagnostic accuracy. The accuracy index to be maximized is the area or partial area under the receiver operating characteristic (ROC) curve. We propose a distribution-free rank-based approach for optimizing the area under the ROC curve and compare it with logistic regression and with classic linear discriminant analysis (LDA). It has been shown that the latter method optimizes the area under the ROC curve when test results have a multivariate normal distribution for diseased and non-diseased populations. Simulation studies suggest that the proposed non-parametric method is efficient when data are multivariate normal.The distribution-free method is generalized to a smooth distribution-free approach to: (i) accommodate some reasonable smoothness assumptions; (ii) incorporate covariate effects; and (iii) yield optimized partial areas under the ROC curve. This latter feature is particularly important since it allows one to focus on a region of the ROC curve which is of most relevance to clinical practice. Neither logistic regression nor LDA necessarily maximize partial areas. The approaches are illustrated on two cancer datasets, one involving serum antigen markers for pancreatic cancer and the other involving longitudinal prostate specific antigen data.     

Time‐dependent classification accuracy curve under marker‐dependent sampling

Evaluating the classification accuracy of a candidate biomarker signaling the onset of disease or disease status is essential for medical decision making. A good biomarker would accurately identify the patients who are likely to progress or die at a particular time in the future or who are in urgent need for active treatments. To assess the performance of a candidate biomarker, the receiver operating characteristic (ROC) curve and the area under the ROC curve (AUC) are commonly used. In many cases, the standard simple random sampling (SRS) design used for biomarker validation studies is costly and inefficient. In order to improve the efficiency and reduce the cost of biomarker validation, marker‐dependent sampling (MDS) may be used. In a MDS design, the selection of patients to assess true survival time is dependent on the result of a biomarker assay. In this article, we introduce a nonparametric estimator for time‐dependent AUC under a MDS design. The consistency and the asymptotic normality of the proposed estimator is established. Simulation shows the unbiasedness of the proposed estimator and a significant efficiency gain of the MDS design over the SRS design.     

A Semiparametric Joint Model for Longitudinal and Survival Data with Application to Hemodialysis Study

Summary .  In many longitudinal clinical studies, the level and progression rate of repeatedly measured biomarkers on each subject quantify the severity of the disease and that subject's susceptibility to progression of the disease. It is of scientific and clinical interest to relate such quantities to a later time-to-event clinical endpoint such as patient survival. This is usually done with a shared parameter model. In such models, the longitudinal biomarker data and the survival outcome of each subject are assumed to be conditionally independent given subject-level severity or susceptibility (also called frailty in statistical terms). In this article, we study the case where the conditional distribution of longitudinal data is modeled by a linear mixed-effect model, and the conditional distribution of the survival data is given by a Cox proportional hazard model. We allow unknown regression coefficients and time-dependent covariates in both models. The proposed estimators are maximizers of an exact correction to the joint log likelihood with the frailties eliminated as nuisance parameters, an idea that originated from correction of covariate measurement error in measurement error models. The corrected joint log likelihood is shown to be asymptotically concave and leads to consistent and asymptotically normal estimators. Unlike most published methods for joint modeling, the proposed estimation procedure does not rely on distributional assumptions of the frailties. The proposed method was studied in simulations and applied to a data set from the Hemodialysis Study.     

Construction of confidence intervals for the maximum of the Youden index and the corresponding cutoff point of a continuous biomarker

Evaluation of the overall accuracy of biomarkers might be based on average measures of the sensitivity for all possible specificities ‐and vice versa‐ or equivalently the area under the receiver operating characteristic (ROC) curve that is typically used in such settings. In practice clinicians are in need of a cutoff point to determine whether intervention is required after establishing the utility of a continuous biomarker. The Youden index can serve both purposes as an overall index of a biomarker's accuracy, that also corresponds to an optimal, in terms of maximizing the Youden index, cutoff point that in turn can be utilized for decision making. In this paper, we provide new methods for constructing confidence intervals for both the Youden index and its corresponding cutoff point. We explore approaches based on the delta approximation under the normality assumption, as well as power transformations to normality and nonparametric kernel‐ and spline‐based approaches. We compare our methods to existing techniques through simulations in terms of coverage and width. We then apply the proposed methods to serum‐based markers of a prospective observational study involving diagnosis of late‐onset sepsis in neonates.