Minimum‐Norm Estimation for Binormal Receiver Operating Characteristic (ROC) Curves

The receiver operating characteristic (ROC) curve is often used to assess the usefulness of a diagnostic test. We present a new method to estimate the parameters of a popular semi‐parametric ROC model, called the binormal model. Our method is based on minimization of the functional distance between two estimators of an unknown transformation postulated by the model, and has a simple, closed‐form solution. We study the asymptotics of our estimators, show via simulation that they compare favorably with existing estimators, and illustrate how covariates may be incorporated into the norm minimization framework.     

Inverse probability weighting estimation of the volume under the ROC surface in the presence of verification bias

In diagnostic medicine, the volume under the receiver operating characteristic (ROC) surface (VUS) is a commonly used index to quantify the ability of a continuous diagnostic test to discriminate between three disease states. In practice, verification of the true disease status may be performed only for a subset of subjects under study since the verification procedure is invasive, risky, or expensive. The selection for disease examination might depend on the results of the diagnostic test and other clinical characteristics of the patients, which in turn can cause bias in estimates of the VUS. This bias is referred to as verification bias. Existing verification bias correction in three‐way ROC analysis focuses on ordinal tests. We propose verification bias‐correction methods to construct ROC surface and estimate the VUS for a continuous diagnostic test, based on inverse probability weighting. By applying U‐statistics theory, we develop asymptotic properties for the estimator. A Jackknife estimator of variance is also derived. Extensive simulation studies are performed to evaluate the performance of the new estimators in terms of bias correction and variance. The proposed methods are used to assess the ability of a biomarker to accurately identify stages of Alzheimer's disease.     

Spind: a package for computing spatially corrected accuracy measures

Using an appropriate accuracy measure is essential for assessing prediction accuracy in species distribution modelling. Therefore, model evaluation as an analytical uncertainty is a challenging problem. Although a variety of accuracy measures for the assessment of prediction errors in presence/absence models is available, there is a lack of spatial accuracy measures, i.e. measures that are sensitive to the spatial arrangement of the predictions. We present ‘spind’, a new software package (based on the R software program) that provides spatial performance measures for grid‐based models. These accuracy measures are generalized, spatially corrected versions of the classical ones, thus enabling comparisons between them. Our method for evaluation consists of the following steps: 1) incorporate additional autocorrelation until spatial autocorrelation in predictions and actuals is balanced, 2) cross‐classify predictions and adjusted actuals in a 4 × 4 contingency table, 3) use a refined weighting pattern for errors, and 4) calculate weighted Kappa, sensitivity, specificity and subsequently ROC, AUC, TSS to get spatially corrected indices. To illustrate the impact of our spatial method we present an example of simulated data as well as an example of presence/absence data of the plant species Dianthus carthusianorum across Germany. Our analysis includes a statistic for the comparison of spatial and classical (non‐spatial) indices. We find that our spatial indices tend to result in higher values than classical ones. These differences are statistically significant at medium and high autocorrelation levels. We conclude that these spatial accuracy measures may contribute to evaluate prediction errors in presence/absence models, especially in case of medium or high degree of similarity of adjacent data, i.e. aggregated (clumped) or continuous species distributions.     

Exploration of Analysis Methods for Diagnostic Imaging Tests: Problems with ROC AUC and Confidence Scores in CT Colonography

Background

Different methods of evaluating diagnostic performance when comparing diagnostic tests may lead to different results. We compared two such approaches, sensitivity and specificity with area under the Receiver Operating Characteristic Curve (ROC AUC) for the evaluation of CT colonography for the detection of polyps, either with or without computer assisted detection.

Methods

In a multireader multicase study of 10 readers and 107 cases we compared sensitivity and specificity, using radiological reporting of the presence or absence of polyps, to ROC AUC calculated from confidence scores concerning the presence of polyps. Both methods were assessed against a reference standard. Here we focus on five readers, selected to illustrate issues in design and analysis. We compared diagnostic measures within readers, showing that differences in results are due to statistical methods.

Results

Reader performance varied widely depending on whether sensitivity and specificity or ROC AUC was used. There were problems using confidence scores; in assigning scores to all cases; in use of zero scores when no polyps were identified; the bimodal non-normal distribution of scores; fitting ROC curves due to extrapolation beyond the study data; and the undue influence of a few false positive results. Variation due to use of different ROC methods exceeded differences between test results for ROC AUC.

Conclusions

The confidence scores recorded in our study violated many assumptions of ROC AUC methods, rendering these methods inappropriate. The problems we identified will apply to other detection studies using confidence scores. We found sensitivity and specificity were a more reliable and clinically appropriate method to compare diagnostic tests.     

Estimation of the ROC Curve under Verification Bias

The ROC (receiver operating characteristic) curve is the most commonly used statistical tool for describing the discriminatory accuracy of a diagnostic test. Classical estimation of the ROC curve relies on data from a simple random sample from the target population. In practice, estimation is often complicated due to not all subjects undergoing a definitive assessment of disease status (verification). Estimation of the ROC curve based on data only from subjects with verified disease status may be badly biased. In this work we investigate the properties of the doubly robust (DR) method for estimating the ROC curve under verification bias originally developed by Rotnitzky, Faraggi and Schisterman (2006) for estimating the area under the ROC curve. The DR method can be applied for continuous scaled tests and allows for a non‐ignorable process of selection to verification. We develop the estimator's asymptotic distribution and examine its finite sample properties via a simulation study. We exemplify the DR procedure for estimation of ROC curves with data collected on patients undergoing electron beam computer tomography, a diagnostic test for calcification of the arteries.     

Binding interface prediction by combining protein–protein docking results

We developed a method called residue contact frequency (RCF), which uses the complex structures generated by the protein–protein docking algorithm ZDOCK to predict interface residues. Unlike interface prediction algorithms that are based on monomers alone, RCF is binding partner specific. We evaluated the performance of RCF using the area under the precision‐recall (PR) curve (AUC) on a large protein docking Benchmark. RCF (AUC = 0.44) performed as well as meta‐PPISP (AUC = 0.43), which is one of the best monomer‐based interface prediction methods. In addition, we test a support vector machine (SVM) to combine RCF with meta‐PPISP and another monomer‐based interface prediction algorithm Evolutionary Trace to further improve the performance. We found that the SVM that combined RCF and meta‐PPISP achieved the best performance (AUC = 0.47). We used RCF to predict the binding interfaces of proteins that can bind to multiple partners and RCF was able to correctly predict interface residues that are unique for the respective binding partners. Furthermore, we found that residues that contributed greatly to binding affinity (hotspot residues) had significantly higher RCF than other residues. Proteins 2014; 82:57–66. © 2013 Wiley Periodicals, Inc.     

Serum miR‐92a‐1 is a novel diagnostic biomarker for colorectal cancer

Colorectal cancer (CRC) is one of the most common cancers worldwide, with high mortality. Abnormally expressed microRNAs (miRNAs) are considered novel biomarkers in cancer diagnosis. The aim of this study was to investigate the diagnostic value of miR‐92a‐1 in patients with CRC. Serum samples were collected from 148 patients pathologically diagnosed with CRC and 68 gender‐ and age‐matched healthy volunteers. Quantitative real‐time polymerase chain reaction (qRT‐PCR) was used to measure serum miR‐92a‐1 level. Relationship between miR‐92a‐1 and clinicopathological features of CRC cases was analysed via chi‐square test. Receiver operating characteristic (ROC) curve was plotted to estimate the diagnostic value of miR‐92a‐1 in CRC. Serum miR‐92a‐1 was significantly up‐regulated in CRC patients compared with healthy individuals (P < .001). Moreover, miR‐92a‐1 expression was correlated with TNM stage (P = .02), histological stage (P = .003), lymph node metastasis (P = .003) and distant metastasis (P < .001). ROC analysis showed that the area under the ROC curve (AUC) was 0.914, suggesting high diagnostic accuracy of miR‐92a‐1 in ROC. The optimal cut‐off value was 1.485, with a sensitivity of 81.8% and a specificity of 95.6%. MiR‐92a‐1 is increased in CRC patients and correlated with aggressive clinical characteristics. Serum miR‐92a‐1 may be a potential diagnostic biomarker for CRC.     

Reproducibility in urine peptidome profiling using MALDI‐TOF

MALDI‐TOF profiling of low molecular weight peptides (peptidome) usage is limited due to the lack of reproducibility from the confounding inferences of sample preparation, data acquisition, and processing. We applied MALDI‐TOF analysis to profile urine peptidome with the aims to: (i) compare centrifugal ultrafiltration and dialysis pretreatments, (ii) determine whether using signal LOD (sLOD), together with data normalization, may reduce MALDI‐TOF variability. We also investigated the influence of peaks detection on reproducibility. Dialysis allowed to obtain better MALDI‐TOF spectra than ultrafiltration. Within the 1000–4000 m/z range, we identified 120 and 129 peaks in intra‐ and interassay studies, respectively. To estimate the sLOD, serial dilution of pooled urines up to 1/256 were analyzed in triplicate. Six data normalization strategies were investigated–the mean, median, internal standard, relative intensity, TIC, and linear rescaling normalization. Normalization methods alone performed poorly in reducing features variability while when combined to sLOD adjustment showed an overall reduction in features CVs. Applying a feedback signal processing approach, after median normalization and sLOD adjustment, CVs were reduced from 103 to 26% and 113 to 25% for the intra‐ and interassay, respectively, and spectra became more comparable in terms of data dispersion.     

Testing for Treatment Effects on Subsets of Endpoints

Multiple endpoints are tested to assess an overall treatment effect and also to identify which endpoints or subsets of endpoints contributed to treatment differences. The conventional p‐value adjustment methods, such as single‐step, step‐up, or step‐down procedures, sequentially identify each significant individual endpoint. Closed test procedures can also detect individual endpoints that have effects via a step‐by‐step closed strategy. This paper proposes a global‐based statistic for testing an a priori number, say, r of the k endpoints, as opposed to the conventional approach of testing one (r = 1) endpoint. The proposed test statistic is an extension of the single‐step p‐value‐based statistic based on the distribution of the smallest p‐value. The test maintains strong control of the FamilyWise Error (FWE) rate under the null hypothesis of no difference in any (sub)set of r endpoints among all possible combinations of the k endpoints. After rejecting the null hypothesis, the individual endpoints in the sets that are rejected can be tested further, using a univariate test statistic in a second step, if desired. However, the second step test only weakly controls the FWE. The proposed method is illustrated by application to a psychosis data set.     

A five‐microRNA signature for individualized prognosis evaluation and radiotherapy guidance in patients with diffuse lower‐grade glioma

Despite the prognostic value of IDH and other gene mutations found in diffuse glioma, markers that judge individual prognosis of patients with diffuse lower‐grade glioma (LGG) are still lacking. This study aims to develop an expression‐based microRNA signature to provide survival and radiotherapeutic response prediction for LGG patients. MicroRNA expression profiles and relevant clinical information of LGG patients were downloaded from The Cancer Genome Atlas (TCGA; the training group) and the Chinese Glioma Genome Atlas (CGGA; the test group). Cox regression analysis, random survival forests‐variable hunting (RSFVH) screening and receiver operating characteristic (ROC) were used to identify the prognostic microRNA signature. ROC and TimeROC curves were plotted to compare the predictive ability of IDH mutation and the signature. Stratification analysis was conducted in patients with radiotherapy information. Gene ontology (GO) analysis and Kyoto Encyclopedia of Genes and Genomes (KEGG) pathway analysis were performed to explore the biological function of the signature. We identified a five‐microRNA signature that can classify patients into low‐risk or high‐risk group with significantly different survival in the training and test datasets (P < 0.001). The five‐microRNA signature was proved to be superior to IDH mutation in survival prediction (AUCtraining = 0.688 vs 0.607). Stratification analysis found the signature could further divide patients after radiotherapy into two risk groups. GO and KEGG analyses revealed that microRNAs from the prognostic signature were mainly enriched in cancer‐associated pathways. The newly discovered five‐microRNA signature could predict survival and radiotherapeutic response of LGG patients based on individual microRNA expression.     

Unbiased and Locally Efficient Estimation of Genetic Effect on Quantitative Trait in the Presence of Population Admixture

Summary Population admixture can be a confounding factor in genetic association studies. Family‐based methods ( Rabinowitz and Larid, 2000 , Human Heredity 50, 211–223) have been proposed in both testing and estimation settings to adjust for this confounding, especially in case‐only association studies. The family‐based methods rely on conditioning on the observed parental genotypes or on the minimal sufficient statistic for the genetic model under the null hypothesis. In some cases, these methods do not capture all the available information due to the conditioning strategy being too stringent. General efficient methods to adjust for population admixture that use all the available information have been proposed ( Rabinowitz, 2002 , Journal of the American Statistical Association 92, 742–758). However these approaches may not be easy to implement in some situations. A previously developed easy‐to‐compute approach adjusts for admixture by adding supplemental covariates to linear models ( Yang et al., 2000 , Human Heredity 50, 227–233). Here is shown that this augmenting linear model with appropriate covariates strategy can be combined with the general efficient methods in Rabinowitz (2002) to provide computationally tractable and locally efficient adjustment. After deriving the optimal covariates, the adjusted analysis can be carried out using standard statistical software packages such as SAS or R . The proposed methods enjoy a local efficiency in a neighborhood of the true model. The simulation studies show that nontrivial efficiency gains can be obtained by using information not accessible to the methods that rely on conditioning on the minimal sufficient statistics. The approaches are illustrated through an analysis of the influence of apolipoprotein E (APOE) genotype on plasma low‐density lipoprotein (LDL) concentration in children.     

Pooling biospecimens and limits of detection: effects on ROC curve analysis

Frequently, epidemiological studies deal with two restrictions in the evaluation of biomarkers: cost and instrument sensitivity. Costs can hamper the evaluation of the effectiveness of new biomarkers. In addition, many assays are affected by a limit of detection (LOD), depending on the instrument sensitivity. Two common strategies used to cut costs include taking a random sample of the available samples and pooling biospecimens. We compare the two sampling strategies when an LOD effect exists. These strategies are compared by examining the efficiency of receiver operating characteristic (ROC) curve analysis, specifically the estimation of the area under the ROC curve (AUC) for normally distributed markers. We propose and examine a method to estimate AUC when dealing with data from pooled and unpooled samples where an LOD is in effect. In conclusion, pooling is the most efficient cost-cutting strategy when the LOD affects less than 50% of the data. However, when much more than 50% of the data are affected, utilization of the pooling design is not recommended.     

Partial AUC estimation and regression

Accurate diagnosis of disease is a critical part of health care. New diagnostic and screening tests must be evaluated based on their abilities to discriminate diseased from nondiseased states. The partial area under the receiver operating characteristic (ROC) curve is a measure of diagnostic test accuracy. We present an interpretation of the partial area under the curve (AUC), which gives rise to a nonparametric estimator. This estimator is more robust than existing estimators, which make parametric assumptions. We show that the robustness is gained with only a moderate loss in efficiency. We describe a regression modeling framework for making inference about covariate effects on the partial AUC. Such models can refine knowledge about test accuracy. Model parameters can be estimated using binary regression methods. We use the regression framework to compare two prostate-specific antigen biomarkers and to evaluate the dependence of biomarker accuracy on the time prior to clinical diagnosis of prostate cancer.     

Robust Estimation of Area Under ROC Curve Using Auxiliary Variables in the Presence of Missing Biomarker Values

Summary In medical research, the receiver operating characteristic (ROC) curves can be used to evaluate the performance of biomarkers for diagnosing diseases or predicting the risk of developing a disease in the future. The area under the ROC curve (ROC AUC), as a summary measure of ROC curves, is widely utilized, especially when comparing multiple ROC curves. In observational studies, the estimation of the AUC is often complicated by the presence of missing biomarker values, which means that the existing estimators of the AUC are potentially biased. In this article, we develop robust statistical methods for estimating the ROC AUC and the proposed methods use information from auxiliary variables that are potentially predictive of the missingness of the biomarkers or the missing biomarker values. We are particularly interested in auxiliary variables that are predictive of the missing biomarker values. In the case of missing at random (MAR), that is, missingness of biomarker values only depends on the observed data, our estimators have the attractive feature of being consistent if one correctly specifies, conditional on auxiliary variables and disease status, either the model for the probabilities of being missing or the model for the biomarker values. In the case of missing not at random (MNAR), that is, missingness may depend on the unobserved biomarker values, we propose a sensitivity analysis to assess the impact of MNAR on the estimation of the ROC AUC. The asymptotic properties of the proposed estimators are studied and their finite‐sample behaviors are evaluated in simulation studies. The methods are further illustrated using data from a study of maternal depression during pregnancy.     

Comprehensive analysis of autophagy‐related prognostic genes in breast cancer

Accumulating evidence revealed that autophagy played vital roles in breast cancer (BC) progression. Thus, the aim of this study was to investigate the prognostic value of autophagy‐related genes (ARGs) and develop a ARG‐based model to evaluate 5‐year overall survival (OS) in BC patients. We acquired ARG expression profiling in a large BC cohort (N = 1007) from The Cancer Genome Atlas (TCGA) database. The correlation between ARGs and OS was confirmed by the LASSO and Cox regression analyses. A predictive model was established based on independent prognostic variables. Thus, time‐dependent receiver operating curve (ROC), calibration plot, decision curve and subgroup analysis were conducted to determine the predictive performance of ARG‐based model. Four ARGs (ATG4A, IFNG, NRG1 and SERPINA1) were identified using the LASSO and multivariate Cox regression analyses. A ARG‐based model was constructed based on the four ARGs and two clinicopathological risk factors (age and TNM stage), dividing patients into high‐risk and low‐risk groups. The 5‐year OS of patients in the low‐risk group was higher than that in the high‐risk group (P < 0.0001). Time‐dependent ROC at 5 years indicated that the four ARG–based tool had better prognostic accuracy than TNM stage in the training cohort (AUC: 0.731 vs 0.640, P < 0.01) and validation cohort (AUC: 0.804 vs 0.671, P < 0.01). The mutation frequencies of the four ARGs (ATG4A, IFNG, NRG1 and SERPINA1) were 0.9%, 2.8%, 8% and 1.3%, respectively. We built and verified a novel four ARG–based nomogram, a credible approach to predict 5‐year OS in BC, which can assist oncologists in determining effective therapeutic strategies.     

Ecological niche modeling re‐examined: A case study with the Darwin's fox

Many previous studies have attempted to assess ecological niche modeling performance using receiver operating characteristic (ROC) approaches, even though diverse problems with this metric have been pointed out in the literature. We explored different evaluation metrics based on independent testing data using the Darwin's Fox (Lycalopex fulvipes) as a detailed case in point. Six ecological niche models (ENMs; generalized linear models, boosted regression trees, Maxent, GARP, multivariable kernel density estimation, and NicheA) were explored and tested using six evaluation metrics (partial ROC, Akaike information criterion, omission rate, cumulative binomial probability), including two novel metrics to quantify model extrapolation versus interpolation (E‐space index I) and extent of extrapolation versus Jaccard similarity (E‐space index II). Different ENMs showed diverse and mixed performance, depending on the evaluation metric used. Because ENMs performed differently according to the evaluation metric employed, model selection should be based on the data available, assumptions necessary, and the particular research question. The typical ROC AUC evaluation approach should be discontinued when only presence data are available, and evaluations in environmental dimensions should be adopted as part of the toolkit of ENM researchers. Our results suggest that selecting Maxent ENM based solely on previous reports of its performance is a questionable practice. Instead, model comparisons, including diverse algorithms and parameterizations, should be the sine qua non for every study using ecological niche modeling. ENM evaluations should be developed using metrics that assess desired model characteristics instead of single measurement of fit between model and data. The metrics proposed herein that assess model performance in environmental space (i.e., E‐space indices I and II) may complement current methods for ENM evaluation.     

Inverse probability weighting methods for Cox regression with right-truncated data

Right-truncated data arise when observations are ascertained retrospectively, and only subjects who experience the event of interest by the time of sampling are selected. Such a selection scheme, without adjustment, leads to biased estimation of covariate effects in the Cox proportional hazards model. The existing methods for fitting the Cox model to right-truncated data, which are based on the maximization of the likelihood or solving estimating equations with respect to both the baseline hazard function and the covariate effects, are numerically challenging. We consider two alternative simple methods based on inverse probability weighting (IPW) estimating equations, which allow consistent estimation of covariate effects under a positivity assumption and avoid estimation of baseline hazards. We discuss problems of identifiability and consistency that arise when positivity does not hold and show that although the partial tests for null effects based on these IPW methods can be used in some settings even in the absence of positivity, they are not valid in general. We propose adjusted estimating equations that incorporate the probability of observation when it is known from external sources, which results in consistent estimation. We compare the methods in simulations and apply them to the analyses of human immunodeficiency virus latency.