The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets

Binary classifiers are routinely evaluated with performance measures such as sensitivity and specificity, and performance is frequently illustrated with Receiver Operating Characteristics (ROC) plots. Alternative measures such as positive predictive value (PPV) and the associated Precision/Recall (PRC) plots are used less frequently. Many bioinformatics studies develop and evaluate classifiers that are to be applied to strongly imbalanced datasets in which the number of negatives outweighs the number of positives significantly. While ROC plots are visually appealing and provide an overview of a classifier''s performance across a wide range of specificities, one can ask whether ROC plots could be misleading when applied in imbalanced classification scenarios. We show here that the visual interpretability of ROC plots in the context of imbalanced datasets can be deceptive with respect to conclusions about the reliability of classification performance, owing to an intuitive but wrong interpretation of specificity. PRC plots, on the other hand, can provide the viewer with an accurate prediction of future classification performance due to the fact that they evaluate the fraction of true positives among positive predictions. Our findings have potential implications for the interpretation of a large number of studies that use ROC plots on imbalanced datasets.     

Ecological niche modeling re‐examined: A case study with the Darwin's fox

Many previous studies have attempted to assess ecological niche modeling performance using receiver operating characteristic (ROC) approaches, even though diverse problems with this metric have been pointed out in the literature. We explored different evaluation metrics based on independent testing data using the Darwin's Fox (Lycalopex fulvipes) as a detailed case in point. Six ecological niche models (ENMs; generalized linear models, boosted regression trees, Maxent, GARP, multivariable kernel density estimation, and NicheA) were explored and tested using six evaluation metrics (partial ROC, Akaike information criterion, omission rate, cumulative binomial probability), including two novel metrics to quantify model extrapolation versus interpolation (E‐space index I) and extent of extrapolation versus Jaccard similarity (E‐space index II). Different ENMs showed diverse and mixed performance, depending on the evaluation metric used. Because ENMs performed differently according to the evaluation metric employed, model selection should be based on the data available, assumptions necessary, and the particular research question. The typical ROC AUC evaluation approach should be discontinued when only presence data are available, and evaluations in environmental dimensions should be adopted as part of the toolkit of ENM researchers. Our results suggest that selecting Maxent ENM based solely on previous reports of its performance is a questionable practice. Instead, model comparisons, including diverse algorithms and parameterizations, should be the sine qua non for every study using ecological niche modeling. ENM evaluations should be developed using metrics that assess desired model characteristics instead of single measurement of fit between model and data. The metrics proposed herein that assess model performance in environmental space (i.e., E‐space indices I and II) may complement current methods for ENM evaluation.     

Adjusted geometric-mean: a novel performance measure for imbalanced bioinformatics datasets learning

One common and challenging problem faced by many bioinformatics applications, such as promoter recognition, splice site prediction, RNA gene prediction, drug discovery and protein classification, is the imbalance of the available datasets. In most of these applications, the positive data examples are largely outnumbered by the negative data examples, which often leads to the development of sub-optimal prediction models having high negative recognition rate (Specificity = SP) and low positive recognition rate (Sensitivity = SE). When class imbalance learning methods are applied, usually, the SE is increased at the expense of reducing some amount of the SP. In this paper, we point out that in these data-imbalanced bioinformatics applications, the goal of applying class imbalance learning methods would be to increase the SE as high as possible by keeping the reduction of SP as low as possible. We explain that the existing performance measures used in class imbalance learning can still produce sub-optimal models with respect to this classification goal. In order to overcome these problems, we introduce a new performance measure called Adjusted Geometric-mean (AGm). The experimental results obtained on ten real-world imbalanced bioinformatics datasets demonstrates that the AGm metric can achieve a lower rate of reduction of SP than the existing performance metrics, when increasing the SE through class imbalance learning methods. This characteristic of AGm metric makes it more suitable for achieving the proposed classification goal in imbalanced bioinformatics datasets learning.     

Imbalanced biomedical data classification using self-adaptive multilayer ELM combined with dynamic GAN

Background

Imbalanced data classification is an inevitable problem in medical intelligent diagnosis. Most of real-world biomedical datasets are usually along with limited samples and high-dimensional feature. This seriously affects the classification performance of the model and causes erroneous guidance for the diagnosis of diseases. Exploring an effective classification method for imbalanced and limited biomedical dataset is a challenging task.

Methods

In this paper, we propose a novel multilayer extreme learning machine (ELM) classification model combined with dynamic generative adversarial net (GAN) to tackle limited and imbalanced biomedical data. Firstly, principal component analysis is utilized to remove irrelevant and redundant features. Meanwhile, more meaningful pathological features are extracted. After that, dynamic GAN is designed to generate the realistic-looking minority class samples, thereby balancing the class distribution and avoiding overfitting effectively. Finally, a self-adaptive multilayer ELM is proposed to classify the balanced dataset. The analytic expression for the numbers of hidden layer and node is determined by quantitatively establishing the relationship between the change of imbalance ratio and the hyper-parameters of the model. Reducing interactive parameters adjustment makes the classification model more robust.

Results

To evaluate the classification performance of the proposed method, numerical experiments are conducted on four real-world biomedical datasets. The proposed method can generate authentic minority class samples and self-adaptively select the optimal parameters of learning model. By comparing with W-ELM, SMOTE-ELM, and H-ELM methods, the quantitative experimental results demonstrate that our method can achieve better classification performance and higher computational efficiency in terms of ROC, AUC, G-mean, and F-measure metrics.

Conclusions

Our study provides an effective solution for imbalanced biomedical data classification under the condition of limited samples and high-dimensional feature. The proposed method could offer a theoretical basis for computer-aided diagnosis. It has the potential to be applied in biomedical clinical practice.

     

Automated analysis of high‐content microscopy data with deep learning

Existing computational pipelines for quantitative analysis of high‐content microscopy data rely on traditional machine learning approaches that fail to accurately classify more than a single dataset without substantial tuning and training, requiring extensive analysis. Here, we demonstrate that the application of deep learning to biological image data can overcome the pitfalls associated with conventional machine learning classifiers. Using a deep convolutional neural network (DeepLoc) to analyze yeast cell images, we show improved performance over traditional approaches in the automated classification of protein subcellular localization. We also demonstrate the ability of DeepLoc to classify highly divergent image sets, including images of pheromone‐arrested cells with abnormal cellular morphology, as well as images generated in different genetic backgrounds and in different laboratories. We offer an open‐source implementation that enables updating DeepLoc on new microscopy datasets. This study highlights deep learning as an important tool for the expedited analysis of high‐content microscopy data.     

AVC: Selecting discriminative features on basis of AUC by maximizing variable complementarity

Background

The Receiver Operator Characteristic (ROC) curve is well-known in evaluating classification performance in biomedical field. Owing to its superiority in dealing with imbalanced and cost-sensitive data, the ROC curve has been exploited as a popular metric to evaluate and find out disease-related genes (features). The existing ROC-based feature selection approaches are simple and effective in evaluating individual features. However, these approaches may fail to find real target feature subset due to their lack of effective means to reduce the redundancy between features, which is essential in machine learning.

Results

In this paper, we propose to assess feature complementarity by a trick of measuring the distances between the misclassified instances and their nearest misses on the dimensions of pairwise features. If a misclassified instance and its nearest miss on one feature dimension are far apart on another feature dimension, the two features are regarded as complementary to each other. Subsequently, we propose a novel filter feature selection approach on the basis of the ROC analysis. The new approach employs an efficient heuristic search strategy to select optimal features with highest complementarities. The experimental results on a broad range of microarray data sets validate that the classifiers built on the feature subset selected by our approach can get the minimal balanced error rate with a small amount of significant features.

Conclusions

Compared with other ROC-based feature selection approaches, our new approach can select fewer features and effectively improve the classification performance.

     

Efficient approximate k‐fold and leave‐one‐out cross‐validation for ridge regression

In model building and model evaluation, cross‐validation is a frequently used resampling method. Unfortunately, this method can be quite time consuming. In this article, we discuss an approximation method that is much faster and can be used in generalized linear models and Cox’ proportional hazards model with a ridge penalty term. Our approximation method is based on a Taylor expansion around the estimate of the full model. In this way, all cross‐validated estimates are approximated without refitting the model. The tuning parameter can now be chosen based on these approximations and can be optimized in less time. The method is most accurate when approximating leave‐one‐out cross‐validation results for large data sets which is originally the most computationally demanding situation. In order to demonstrate the method's performance, it will be applied to several microarray data sets. An R package penalized, which implements the method, is available on CRAN.     

Development and testing of a genetic marker‐based pedigree reconstruction system ‘PR‐genie’ incorporating size‐class data

For wildlife populations, it is often difficult to determine biological parameters that indicate breeding patterns and population mixing, but knowledge of these parameters is essential for effective management. A pedigree encodes the relationship between individuals and can provide insight into the dynamics of a population over its recent history. Here, we present a method for the reconstruction of pedigrees for wild populations of animals that live long enough to breed multiple times over their lifetime and that have complex or unknown generational structures. Reconstruction was based on microsatellite genotype data along with ancillary biological information: sex and observed body size class as an indicator of relative age of individuals within the population. Using body size‐class data to infer relative age has not been considered previously in wildlife genealogy and provides a marked improvement in accuracy of pedigree reconstruction. Body size‐class data are particularly useful for wild populations because it is much easier to collect noninvasively than absolute age data. This new pedigree reconstruction system, PR‐genie, performs reconstruction using maximum likelihood with optimization driven by the cross‐entropy method. We demonstrated pedigree reconstruction performance on simulated populations (comparing reconstructed pedigrees to known true pedigrees) over a wide range of population parameters and under assortative and intergenerational mating schema. Reconstruction accuracy increased with the presence of size‐class data and as the amount and quality of genetic data increased. We provide recommendations as to the amount and quality of data necessary to provide insight into detailed familial relationships in a wildlife population using this pedigree reconstruction technique.     

Trait‐based risk assessment for invasive species: high performance across diverse taxonomic groups,geographic ranges and machine learning/statistical tools

Aim Trait‐based risk assessment for invasive species is becoming an important tool for identifying non‐indigenous species that are likely to cause harm. Despite this, concerns remain that the invasion process is too complex for accurate predictions to be made. Our goal was to test risk assessment performance across a range of taxonomic and geographical scales, at different points in the invasion process, with a range of statistical and machine learning algorithms. Location Regional to global data sets. Methods We selected six data sets differing in size, geography and taxonomic scope. For each data set, we created seven risk assessment tools using a range of statistical and machine learning algorithms. Performance of tools was compared to determine the effects of data set size and scale, the algorithm used, and to determine overall performance of the trait‐based risk assessment approach. Results Risk assessment tools with good performance were generated for all data sets. Random forests (RF) and logistic regression (LR) consistently produced tools with high performance. Other algorithms had varied performance. Despite their greater power and flexibility, machine learning algorithms did not systematically outperform statistical algorithms. Geographic scope of the data set, and size of the data set, did not systematically affect risk assessment performance. Main conclusions Across six representative data sets, we were able to create risk assessment tools with high performance. Additional data sets could be generated for other taxonomic groups and regions, and these could support efforts to prevent the arrival of new invaders. Random forests and LR approaches performed well for all data sets and could be used as a standard approach to risk assessment development.     

Power analysis for trend in ordinal cover classes: implications for long‐term vegetation monitoring

Question: We provide a method to calculate the power of ordinal regression models for detecting temporal trends in plant abundance measured as ordinal cover classes. Does power depend on the shape of the unobserved (latent) distribution of percentage cover? How do cover class schemes that differ in the number of categories affect power? Methods: We simulated cover class data by “cutting‐up” a continuous logit‐beta distributed variable using 7‐point and 15‐point cover classification schemes. We used Monte Carlo simulation to estimate power for detecting trends with two ordinal models, proportional odds logistic regression (POM) and logistic regression with cover classes re‐binned into two categories, a model we term an assessment point model (APM). We include a model fit to the logit‐transformed percentage cover data for comparison, which is a latent model. Results: The POM had equal or higher power compared to the APM and latent model, but power varied in complex ways as a function of the assumed latent beta distribution. We discovered that if the latent distribution is skewed, a cover class scheme with more categories might yield higher power to detect trend. Conclusions: Our power analysis method maintains the connection between the observed ordinal cover classes and the unmeasured (latent) percentage cover variable, allowing for a biologically meaningful trend to be defined on the percentage cover scale. Both the shape of the latent beta distribution and the alternative hypothesis should be considered carefully when determining sample size requirements for long‐term vegetation monitoring using cover class measurements.     

Modeling of species distributions with Maxent: new extensions and a comprehensive evaluation

Accurate modeling of geographic distributions of species is crucial to various applications in ecology and conservation. The best performing techniques often require some parameter tuning, which may be prohibitively time‐consuming to do separately for each species, or unreliable for small or biased datasets. Additionally, even with the abundance of good quality data, users interested in the application of species models need not have the statistical knowledge required for detailed tuning. In such cases, it is desirable to use “default settings”, tuned and validated on diverse datasets. Maxent is a recently introduced modeling technique, achieving high predictive accuracy and enjoying several additional attractive properties. The performance of Maxent is influenced by a moderate number of parameters. The first contribution of this paper is the empirical tuning of these parameters. Since many datasets lack information about species absence, we present a tuning method that uses presence‐only data. We evaluate our method on independently collected high‐quality presence‐absence data. In addition to tuning, we introduce several concepts that improve the predictive accuracy and running time of Maxent. We introduce “hinge features” that model more complex relationships in the training data; we describe a new logistic output format that gives an estimate of probability of presence; finally we explore “background sampling” strategies that cope with sample selection bias and decrease model‐building time. Our evaluation, based on a diverse dataset of 226 species from 6 regions, shows: 1) default settings tuned on presence‐only data achieve performance which is almost as good as if they had been tuned on the evaluation data itself; 2) hinge features substantially improve model performance; 3) logistic output improves model calibration, so that large differences in output values correspond better to large differences in suitability; 4) “target‐group” background sampling can give much better predictive performance than random background sampling; 5) random background sampling results in a dramatic decrease in running time, with no decrease in model performance.     

PRC1‐labeled microtubule bundles and kinetochore pairs show one‐to‐one association in metaphase

In the mitotic spindle, kinetochore microtubules form k‐fibers, whereas overlap or interpolar microtubules form antiparallel arrays containing the cross‐linker protein regulator of cytokinesis 1 (PRC1). We have recently shown that an overlap bundle, termed bridging fiber, links outermost sister k‐fibers. However, the relationship between overlap bundles and k‐fibers throughout the spindle remained unknown. Here, we show that in a metaphase spindle more than 90% of overlap bundles act as a bridge between sister k‐fibers. We found that the number of PRC1‐GFP‐labeled bundles per spindle is nearly the same as the number of kinetochore pairs. Live‐cell imaging revealed that kinetochore movement in the equatorial plane of the spindle is highly correlated with the movement of the coupled PRC1‐GFP‐labeled fiber, whereas the correlation with other fibers decreases with increasing distance. Analysis of endogenous PRC1 localization confirmed the results obtained with PRC1‐GFP. PRC1 knockdown reduced the bridging fiber thickness and interkinetochore distance throughout the spindle, suggesting a function of PRC1 in bridging microtubule organization and force balance in the metaphase spindle.     

HPSLPred: An Ensemble Multi‐Label Classifier for Human Protein Subcellular Location Prediction with Imbalanced Source

Predicting the subcellular localization of proteins is an important and challenging problem. Traditional experimental approaches are often expensive and time‐consuming. Consequently, a growing number of research efforts employ a series of machine learning approaches to predict the subcellular location of proteins. There are two main challenges among the state‐of‐the‐art prediction methods. First, most of the existing techniques are designed to deal with multi‐class rather than multi‐label classification, which ignores connections between multiple labels. In reality, multiple locations of particular proteins imply that there are vital and unique biological significances that deserve special focus and cannot be ignored. Second, techniques for handling imbalanced data in multi‐label classification problems are necessary, but never employed. For solving these two issues, we have developed an ensemble multi‐label classifier called HPSLPred, which can be applied for multi‐label classification with an imbalanced protein source. For convenience, a user‐friendly webserver has been established at http://server.malab.cn/HPSLPred.     

ORCHIDEE‐STICS,a process‐based model of sugarcane biomass production: calibration of model parameters governing phenology

Agro‐Land Surface Models (agro‐LSM) combine detailed crop models and large‐scale vegetation models (DGVMs) to model the spatial and temporal distribution of energy, water, and carbon fluxes within the soil–vegetation–atmosphere continuum worldwide. In this study, we identify and optimize parameters controlling leaf area index (LAI) in the agro‐LSM ORCHIDEE‐STICS developed for sugarcane. Using the Morris method to identify the key parameters impacting LAI, at eight different sugarcane field trial sites, in Australia and La Reunion island, we determined that the three most important parameters for simulating LAI are (i) the maximum predefined rate of LAI increase during the early crop development phase, a parameter that defines a plant density threshold below which individual plants do not compete for growing their LAI, and a parameter defining a threshold for nitrogen stress on LAI. A multisite calibration of these three parameters is performed using three different scoring functions. The impact of the choice of a particular scoring function on the optimized parameter values is investigated by testing scoring functions defined from the model‐data RMSE, the figure of merit and a Bayesian quadratic model‐data misfit function. The robustness of the calibration is evaluated for each of the three scoring functions with a systematic cross‐validation method to find the most satisfactory one. Our results show that the figure of merit scoring function is the most robust metric for establishing the best parameter values controlling the LAI. The multisite average figure of merit scoring function is improved from 67% of agreement to 79%. The residual error in LAI simulation after the calibration is discussed.     

Model‐guided design of ligand‐regulated RNAi for programmable control of gene expression

Progress in constructing biological networks will rely on the development of more advanced components that can be predictably modified to yield optimal system performance. We have engineered an RNA‐based platform, which we call an shRNA switch, that provides for integrated ligand control of RNA interference (RNAi) by modular coupling of an aptamer, competing strand, and small hairpin (sh)RNA stem into a single component that links ligand concentration and target gene expression levels. A combined experimental and mathematical modelling approach identified multiple tuning strategies and moves towards a predictable framework for the forward design of shRNA switches. The utility of our platform is highlighted by the demonstration of fine‐tuning, multi‐input control, and model‐guided design of shRNA switches with an optimized dynamic range. Thus, shRNA switches can serve as an advanced component for the construction of complex biological systems and offer a controlled means of activating RNAi in disease therapeutics.     

The effect of fixed‐count subsampling on macroinvertebrate biomonitoring in small streams

• 1 When rigorous standards of collecting and analysing data are maintained, biological monitoring adds valuable information to water resource assessments. Decisions, from study design and field methods to laboratory procedures and data analysis, affect assessment quality. Subsampling ‐ a laboratory procedure in which researchers count and identify a random subset of field samples ‐ is widespread yet controversial. What are the consequences of subsampling?
• 2 To explore this question, random subsamples were computer generated for subsample sizes ranging from 100 to 1000 individuals as compared with the results of counting whole samples. The study was done on benthic invertebrate samples collected from five Puget Sound lowland streams near Seattle, WA, USA. For each replicate subsample, values for 10 biological attributes (e.g. total number of taxa) and for the 10‐metric benthic index of biological integrity (B‐IBI) were computed.
• 3 Variance of each metric and B‐IBI for each subsample size was compared with variance associated with fully counted samples generated using the bootstrap algorithm. From the measures of variance, we computed the maximum number of distinguishable classes of stream condition as a function of sample size for each metric and for B‐IBI.
• 4 Subsampling significantly decreased the maximum number of distinguishable stream classes for B‐IBI, from 8.2 for fully counted samples to 2.8 classes for 100‐organism subsamples. For subsamples containing 100–300 individuals, discriminatory power was low enough to mislead water resource decision makers.

     

A Parametric ROC Model‐Based Approach for Evaluating the Predictiveness of Continuous Markers in Case–Control Studies

Summary The predictiveness curve shows the population distribution of risk endowed by a marker or risk prediction model. It provides a means for assessing the model's capacity for stratifying the population according to risk. Methods for making inference about the predictiveness curve have been developed using cross‐sectional or cohort data. Here we consider inference based on case–control studies, which are far more common in practice. We investigate the relationship between the ROC curve and the predictiveness curve. Insights about their relationship provide alternative ROC interpretations for the predictiveness curve and for a previously proposed summary index of it. Next the relationship motivates ROC based methods for estimating the predictiveness curve. An important advantage of these methods over previously proposed methods is that they are rank invariant. In addition they provide a way of combining information across populations that have similar ROC curves but varying prevalence of the outcome. We apply the methods to prostate‐specific antigen (PSA), a marker for predicting risk of prostate cancer.     

Label‐free diagnosis for colorectal cancer through coffee ring‐assisted surface‐enhanced Raman spectroscopy on blood serum

Surface‐enhanced Raman spectroscopy (SERS) is garnering considerable attention for the swift diagnosis of pathogens and abnormal biological status, that is, cancers. In this work, a simple, fast and inexpensive optical sensing platform is developed by the design of SERS sampling and data analysis. The pretreatment of spectral measurement employed gold nanoparticle colloid mixing with the serum from patients with colorectal cancer (CRC). The droplet of particle‐serum mixture formed coffee‐ring‐like region at the rim, providing strong and stable SERS profiles. The obtained spectra from cancer patients and healthy volunteers were analyzed by unsupervised principal component analysis (PCA) and supervised machine learning model, such as support‐vector machine (SVM), respectively. The results demonstrate that the SVM model provides the superior performance in the classification of CRC diagnosis compared with PCA. In addition, the values of carcinoembryonic antigen from the blood samples were compiled with the corresponding SERS spectra for SVM calculation, yielding improved prediction results.     

Estimating age‐dependent survival from age‐aggregated ringing data—extending the use of historical records

Bird ring‐recovery data have been widely used to estimate demographic parameters such as survival probabilities since the mid‐20th century. However, while the total number of birds ringed each year is usually known, historical information on age at ringing is often not available. A standard ring‐recovery model, for which information on age at ringing is required, cannot be used when historical data are incomplete. We develop a new model to estimate age‐dependent survival probabilities from such historical data when age at ringing is not recorded; we call this the historical data model. This new model provides an extension to the model of Robinson, 2010, Ibis, 152, 651–795 by estimating the proportion of the ringed birds marked as juveniles as an additional parameter. We conduct a simulation study to examine the performance of the historical data model and compare it with other models including the standard and conditional ring‐recovery models. Simulation studies show that the approach of Robinson, 2010, Ibis, 152, 651–795 can cause bias in parameter estimates. In contrast, the historical data model yields similar parameter estimates to the standard model. Parameter redundancy results show that the newly developed historical data model is comparable to the standard ring‐recovery model, in terms of which parameters can be estimated, and has fewer identifiability issues than the conditional model. We illustrate the new proposed model using Blackbird and Sandwich Tern data. The new historical data model allows us to make full use of historical data and estimate the same parameters as the standard model with incomplete data, and in doing so, detect potential changes in demographic parameters further back in time.     

One‐to‐one registration of en‐face optical coherence tomography attenuation coefficients with histology of a prostatectomy specimen

Optical coherence tomography (OCT), enables high‐resolution 3D imaging of the morphology of light scattering tissues. From the OCT signal, parameters can be extracted and related to tissue structures. One of the quantitative parameters is the attenuation coefficient; the rate at which the intensity of detected light decays in depth. To couple the quantitative parameters with the histology one‐to‐one registration is needed. The primary aim of this study is to validate a registration method of quantitative OCT parameters to histological tissue outcome through one‐to‐one registration of OCT with histology. We matched OCT images of unstained fixated prostate tissue slices with corresponding histology slides, wherein different histologic types were demarcated. Attenuation coefficients were determined by a supervised automated exponential fit (corrected for point spread function and sensitivity roll‐off related signal losses) over a depth of 0.32 mm starting from 0.10 mm below the automatically detected tissue edge. Finally, the attenuation coefficients corresponding to the different tissue types of the prostate were compared. From the attenuation coefficients, we produced the squared relative residue and goodness‐of‐fit metric R². This article explains the method to perform supervised automated quantitative analysis of OCT data, and the one‐to‐one registration of OCT extracted quantitative data with histopathological outcomes.