No Longer Confidential: Estimating the Confidence of Individual Regression Predictions

Quantitative predictions in computational life sciences are often based on regression models. The advent of machine learning has led to highly accurate regression models that have gained widespread acceptance. While there are statistical methods available to estimate the global performance of regression models on a test or training dataset, it is often not clear how well this performance transfers to other datasets or how reliable an individual prediction is–a fact that often reduces a user’s trust into a computational method. In analogy to the concept of an experimental error, we sketch how estimators for individual prediction errors can be used to provide confidence intervals for individual predictions. Two novel statistical methods, named CONFINE and CONFIVE, can estimate the reliability of an individual prediction based on the local properties of nearby training data. The methods can be applied equally to linear and non-linear regression methods with very little computational overhead. We compare our confidence estimators with other existing confidence and applicability domain estimators on two biologically relevant problems (MHC–peptide binding prediction and quantitative structure-activity relationship (QSAR)). Our results suggest that the proposed confidence estimators perform comparable to or better than previously proposed estimation methods. Given a sufficient amount of training data, the estimators exhibit error estimates of high quality. In addition, we observed that the quality of estimated confidence intervals is predictable. We discuss how confidence estimation is influenced by noise, the number of features, and the dataset size. Estimating the confidence in individual prediction in terms of error intervals represents an important step from plain, non-informative predictions towards transparent and interpretable predictions that will help to improve the acceptance of computational methods in the biological community.     

What is an expert? A systems perspective on expertise

Expert knowledge is a valuable source of information with a wide range of research applications. Despite the recent advances in defining expert knowledge, little attention has been given to how to view expertise as a system of interacting contributory factors for quantifying an individual's expertise. We present a systems approach to expertise that accounts for many contributing factors and their inter‐relationships and allows quantification of an individual's expertise. A Bayesian network (BN) was chosen for this purpose. For illustration, we focused on taxonomic expertise. The model structure was developed in consultation with taxonomists. The relative importance of the factors within the network was determined by a second set of taxonomists (supra‐experts) who also provided validation of the model structure. Model performance was assessed by applying the model to hypothetical career states of taxonomists designed to incorporate known differences in career states for model testing. The resulting BN model consisted of 18 primary nodes feeding through one to three higher‐order nodes before converging on the target node (Taxonomic Expert). There was strong consistency among node weights provided by the supra‐experts for some nodes, but not others. The higher‐order nodes, “Quality of work” and “Total productivity”, had the greatest weights. Sensitivity analysis indicated that although some factors had stronger influence in the outer nodes of the network, there was relatively equal influence of the factors leading directly into the target node. Despite the differences in the node weights provided by our supra‐experts, there was good agreement among assessments of our hypothetical experts that accurately reflected differences we had specified. This systems approach provides a way of assessing the overall level of expertise of individuals, accounting for multiple contributory factors, and their interactions. Our approach is adaptable to other situations where it is desirable to understand components of expertise.     

Current food web models cannot explain the overall topological structure of observed food webs

Topological food webs illustrating “who eats whom” in different systems exhibit similar, non‐random, structures suggesting that general rules govern food web structure. Current food web models correctly predict many measures of food web topology from knowledge of species richness and connectance (fraction of possible predator–prey links that actually occur), together with assumptions about the ecological rules governing “who eats whom”. However, current measures are relatively insensitive to small changes in topology. Improvement of, and discrimination among, current models requires development of new measures of food web structure. Here I examine whether current food web models (cascade, niche, and nested hierarchy models, plus a random null model) can predict a new measure of food web structure, structural stability. Structural stability complements other measures of food web topology because it is sensitive to changes in topology that other measures often miss. The cascade and null models respectively over‐ and underpredict structural stability for a set of 17 high‐quality food webs. While the niche and nested hierarchy models provide unbiased predictions on average, their 95% confidence intervals frequently fail to include the observed data. Observed structural stabilities for all models are overdispersed compared to model predictions, and predicted and observed structural stabilities are uncorrelated, indicating that important sources of variation in structural stability are not captured by the models. Crucially, poor model performance arises because observed variation in structural stability is unrelated to variation in species richness and connectance. In contrast, almost all other measures of food web topology vary with species richness and connectance in natural webs. No model that takes species richness and connectance as the only input parameters can reproduce observed variation in structural stability. Further progress in predicting and explaining food web topology will require fundamentally new models based on different input parameters.     

Online Prediction of the Running Time of Tasks

We describe and evaluate the Running Time Advisor (RTA), a system that can predict the running time of a compute-bound task on a typical shared, unreserved commodity host. The prediction is computed from linear time series predictions of host load and takes the form of a confidence interval that neatly expresses the error associated with the measurement and prediction processes – error that must be captured to make statistically valid decisions based on the predictions. Adaptive applications make such decisions in pursuit of consistent high performance, choosing, for example, the host where a task is most likely to meet its deadline. We begin by describing the system and summarizing the results of our previously published work on host load prediction. We then describe our algorithm for computing predictions of running time from host load predictions. We next evaluate the system using over 100,000 randomized testcases run on 39 different hosts, finding that is indeed capable of computing correct and useful confidence intervals. Finally, we report on our experience with using the RTA in application-oriented real-time scheduling in distributed systems.     

The Nature of Expertise in Fingerprint Matching: Experts Can Do a Lot with a Little

Expert decision making often seems impressive, even miraculous. People with genuine expertise in a particular domain can perform quickly and accurately, and with little information. In the series of experiments presented here, we manipulate the amount of “information” available to a group of experts whose job it is to identify the source of crime scene fingerprints. In Experiment 1, we reduced the amount of information available to experts by inverting fingerprint pairs and adding visual noise. There was no evidence for an inversion effect—experts were just as accurate for inverted prints as they were for upright prints—but expert performance with artificially noisy prints was impressive. In Experiment 2, we separated matching and nonmatching print pairs in time. Experts were conservative, but they were still able to discriminate pairs of fingerprints that were separated by five-seconds, even though the task was quite different from their everyday experience. In Experiment 3, we separated the print pairs further in time to test the long-term memory of experts compared to novices. Long-term recognition memory for experts and novices was the same, with both performing around chance. In Experiment 4, we presented pairs of fingerprints quickly to experts and novices in a matching task. Experts were more accurate than novices, particularly for similar nonmatching pairs, and experts were generally more accurate when they had more time. It is clear that experts can match prints accurately when there is reduced visual information, reduced opportunity for direct comparison, and reduced time to engage in deliberate reasoning. These findings suggest that non-analytic processing accounts for a substantial portion of the variance in expert fingerprint matching accuracy. Our conclusion is at odds with general wisdom in fingerprint identification practice and formal training, and at odds with the claims and explanations that are offered in court during expert testimony.     

Predictive Assessment of Fish Health and Fish Kills in the Neuse River Estuary Using Elicited Expert Judgment

Declining fish health and the occurrence of large fish kills are some of the more publicly meaningful indicators of water quality in the impaired Neuse River Estuary, North Carolina. It is generally believed that such problems are caused by the widespread depletion of dissolved oxygen—an indirect result of anthropogenic nutrient pollution. However, the development of scientific simulation models to predict how improvements in oxygen conditions will improve the health of fish and reduce the frequency of fish kills has proven elusive. As a pragmatic solution to this problem, the expert opinion of estuarine fisheries scientists in possession of relevant data and experience was elicited. The relations between joint and conditional probabilities were exploited to translate quantities that are normally hard to assess into quantities that can be drawn more directly from the experiential knowledge of the experts. A combined model of expert opinion was constructed as an influence diagram, and Monte Carlo simulation was used to generate predictions of fish health and fish kills in the Neuse River Estuary under current and improved oxygen conditions. Full model results are expressed as probability distributions, capturing the effects of natural variability and knowledge uncertainty—both contributors to total ecological risk.     

Do short courses in evidence based medicine improve knowledge and skills? Validation of Berlin questionnaire and before and after study of courses in evidence based medicine

ObjectiveTo develop and validate an instrument for measuring knowledge and skills in evidence based medicine and to investigate whether short courses in evidence based medicine lead to a meaningful increase in knowledge and skills.DesignDevelopment and validation of an assessment instrument and before and after study.SettingVarious postgraduate short courses in evidence based medicine in Germany.ParticipantsThe instrument was validated with experts in evidence based medicine, postgraduate doctors, and medical students. The effect of courses was assessed by postgraduate doctors from medical and surgical backgrounds.InterventionIntensive 3 day courses in evidence based medicine delivered through tutor facilitated small groups.ResultsThe questionnaire distinguished reliably between groups with different expertise in evidence based medicine. Experts attained a threefold higher average score than students. Postgraduates who had not attended a course performed better than students but significantly worse than experts. Knowledge and skills in evidence based medicine increased after the course by 57% (mean score before course 6.3 (SD 2.9) v 9.9 (SD 2.8), P<0.001). No difference was found among experts or students in absence of an intervention.ConclusionsThe instrument reliably assessed knowledge and skills in evidence based medicine. An intensive 3 day course in evidence based medicine led to a significant increase in knowledge and skills.

What is already known on this topic

Numerous observational studies have investigated the impact of teaching evidence based medicine to healthcare professionals, with conflicting resultsMost of the studies were of poor methodological quality

What this study adds

An instrument assessing basic knowledge and skills required for practising evidence based medicine was developed and validatedAn intensive 3 day course on evidence based medicine for doctors from various backgrounds and training level led to a clinically meaningful improvement of knowledge and skills     

Integrating expert knowledge and ecological niche models to estimate Mexican primates’ distribution

Ecological niche modeling is used to estimate species distributions based on occurrence records and environmental variables, but it seldom includes explicit biotic or historical factors that are important in determining the distribution of species. Expert knowledge can provide additional valuable information regarding ecological or historical attributes of species, but the influence of integrating this information in the modeling process has been poorly explored. Here, we integrated expert knowledge in different stages of the niche modeling process to improve the representation of the actual geographic distributions of Mexican primates (Ateles geoffroyi, Alouatta pigra, and A. palliata mexicana). We designed an elicitation process to acquire information from experts and such information was integrated by an iterative process that consisted of reviews of input data by experts, production of ecological niche models (ENMs), and evaluation of model outputs to provide feedback. We built ENMs using the maximum entropy algorithm along with a dataset of occurrence records gathered from a public source and records provided by the experts. Models without expert knowledge were also built for comparison, and both models, with and without expert knowledge, were evaluated using four validation metrics that provide a measure of accuracy for presence-absence predictions (specificity, sensitivity, kappa, true skill statistic). Integrating expert knowledge to build ENMs produced better results for potential distributions than models without expert knowledge, but a much greater improvement in the transition from potential to realized geographic distributions by reducing overprediction, resulting in better representations of the actual geographic distributions of species. Furthermore, with the combination of niche models and expert knowledge we were able to identify an area of sympatry between A. palliata mexicana and A. pigra. We argue that the inclusion of expert knowledge at different stages in the construction of niche models in an explicit and systematic fashion is a recommended practice as it produces overall positive results for representing realized species distributions.     

Calibration belt for quality-of-care assessment based on dichotomous outcomes

Prognostic models applied in medicine must be validated on independent samples, before their use can be recommended. The assessment of calibration, i.e., the model's ability to provide reliable predictions, is crucial in external validation studies. Besides having several shortcomings, statistical techniques such as the computation of the standardized mortality ratio (SMR) and its confidence intervals, the Hosmer-Lemeshow statistics, and the Cox calibration test, are all non-informative with respect to calibration across risk classes. Accordingly, calibration plots reporting expected versus observed outcomes across risk subsets have been used for many years. Erroneously, the points in the plot (frequently representing deciles of risk) have been connected with lines, generating false calibration curves. Here we propose a methodology to create a confidence band for the calibration curve based on a function that relates expected to observed probabilities across classes of risk. The calibration belt allows the ranges of risk to be spotted where there is a significant deviation from the ideal calibration, and the direction of the deviation to be indicated. This method thus offers a more analytical view in the assessment of quality of care, compared to other approaches.     

草地有害啮齿动物监测专家系统设计介绍

主要介绍了青藏高原草地有害啮齿动物监测专家系统的建造原理和方法。该系统由综合数据库、知识库、推理机、知识编辑语言及系统建造支持环境等部分组成,文中详细说明各部分的主要功能以及与其他部分的相互关系。该系统可对青藏高原有害动物种群动态进行长期监测,快速准确地预测预报草地植被受害状况,并根据生态环境特点,对有害啮齿动物综合治理进行规划,为用户提供长期治理的技术和多种可供选择的可行性方案。     

Estimating parameters of a forest ecosystem C model with measurements of stocks and fluxes as joint constraints

We conducted an inverse modeling analysis, using a variety of data streams (tower-based eddy covariance measurements of net ecosystem exchange, NEE, of CO₂, chamber-based measurements of soil respiration, and ancillary ecological measurements of leaf area index, litterfall, and woody biomass increment) to estimate parameters and initial carbon (C) stocks of a simple forest C-cycle model, DALEC, using Monte Carlo procedures. Our study site is the spruce-dominated Howland Forest AmeriFlux site, in central Maine, USA. Our analysis focuses on: (1) full characterization of data uncertainties, and treatment of these uncertainties in the parameter estimation; (2) evaluation of how combinations of different data streams influence posterior parameter distributions and model uncertainties; and (3) comparison of model performance (in terms of both predicted fluxes and pool dynamics) during a 4-year calibration period (1997–2000) and a 4-year validation period (“forward run”, 2001–2004). We find that woody biomass increment, and, to a lesser degree, soil respiration, measurements contribute to marked reductions in uncertainties in parameter estimates and model predictions as these provide orthogonal constraints to the tower NEE measurements. However, none of the data are effective at constraining fine root or soil C pool dynamics, suggesting that these should be targets for future measurement efforts. A key finding is that adding additional constraints not only reduces uncertainties (i.e., narrower confidence intervals) on model predictions, but at the same time also results in improved model predictions by greatly reducing bias associated with predictions during the forward run.     

Utility of Expert-Based Knowledge for Predicting Wildlife-Vehicle Collisions

ABSTRACT Wildlife-vehicle collisions have important ecological, economic, and social effects. In North America and across northern Europe, moose (Alces alces) are one of the largest ungulates hit by motor vehicles. The force and increasing frequency of these collisions has resulted in a commitment by wildlife and transportation agencies to limit or reduce causal factors. In an effort to improve these mitigation strategies, we used the most readily available source of knowledge of collision factors, expert opinion, to develop a series of models that explained and predicted location of moose-vehicle collisions (MVC). We developed expert-based models using the Analytical Hierarchy Process (AHP) and we used a structured survey approach where experts could assess criteria relevancy, weight criteria, and review weights for consistency. We hypothesized that collisions were the product of habitat- or driver-related factors and we formulated the survey accordingly. We used the receiver operating characteristic to validate the resulting models and the Kappa index of agreement to quantify differences among spatial predictions originating from the experts. Local and nonlocal experts weighted the moose habitat classification as the most important criterion for identifying MVC. Among driver-related criteria, speed limit was weighted as the most important factor. Overall, habitat-based models were more proficient than driver-based models in predicting MVC within Mount Revelstoke and Glacier National Parks, Canada. Both local and nonlocal expert models were excellent predictors of MVC, with local experts slightly outperforming nonlocal experts. Considering that habitat-related criteria were more powerful for predicting MVC, and that habitat can vary considerably across study areas, we suggest that local experts be used when possible. The AHP is a valuable tool for wildlife, highway, and park managers to better understand why and where wildlife-vehicle collisions occur. Adopting this process, our data suggested that MVC were most strongly correlated with highway attractants associated with habitat. Vegetation management or alternative routing could minimize spatial juxtaposition of moose and motor vehicles.     

Reduced Motor Cortex Activity during Movement Preparation following a Period of Motor Skill Practice

Experts in a skill produce movement-related cortical potentials (MRCPs) of smaller amplitude and later onset than novices. This may indicate that, following long-term training, experts require less effort to plan motor skill performance. However, no longitudinal evidence exists to support this claim. To address this, EEG was used to study the effect of motor skill training on cortical activity related to motor planning. Ten non-musicians took part in a 5-week training study learning to play guitar. At week 1, the MRCP was recorded from motor areas whilst participants played the G Major scale. Following a period of practice of the scale, the MRCP was recorded again at week 5. Results showed that the amplitude of the later pre-movement components were smaller at week 5 compared to week 1. This may indicate that, following training, less activity at motor cortex sites is involved in motor skill preparation. This supports claims for a more efficient motor preparation following motor skill training.     

Selective prediction-set models with coverage rate guarantees

The current approach to using machine learning (ML) algorithms in healthcare is to either require clinician oversight for every use case or use their predictions without any human oversight. We explore a middle ground that lets ML algorithms abstain from making a prediction to simultaneously improve their reliability and reduce the burden placed on human experts. To this end, we present a general penalized loss minimization framework for training selective prediction-set (SPS) models, which choose to either output a prediction set or abstain. The resulting models abstain when the outcome is difficult to predict accurately, such as on subjects who are too different from the training data, and achieve higher accuracy on those they do give predictions for. We then introduce a model-agnostic, statistical inference procedure for the coverage rate of an SPS model that ensembles individual models trained using K-fold cross-validation. We find that SPS ensembles attain prediction-set coverage rates closer to the nominal level and have narrower confidence intervals for its marginal coverage rate. We apply our method to train neural networks that abstain more for out-of-sample images on the MNIST digit prediction task and achieve higher predictive accuracy for ICU patients compared to existing approaches.     

Invertebrate traits for the biomonitoring of large European rivers: an assessment of specific types of human impact

1. The power of selected biological invertebrate traits for discriminating different types of human impact (heavy metal pollution and cargo‐ship traffic) were tested using ecological reasoning and linear Discriminant Function Analysis (DFA). 2. Frequency distributions of individual traits and categories of traits from 68 least impacted river reaches (LIRRs) and 304 impacted river reaches were used to define simple assessment rules based on ecological reasoning for specific impairments in large European rivers. In calibration, a maximum of three variables with a priori predictions and two different impairment threshold levels were used. Similarly, DFA was performed on the same variables included in the ecological reasoning approach, but also on all available traits or trait categories. 3. Validation with an independent data set (40 LIRRs, 291 variously impacted river reaches) and using the ecological reasoning approach showed that 75–78% of the reaches were correctly assign with rules on all impact types, 35–57% with rules on heavy metal pollution and 78–93% with rules on cargo‐ship traffic. By comparison, validation showed that DFA performed globally poorer than the ecological reasoning approach. In addition, the performance of the rules based on ecological reasoning remained stable, whereas DFA performance changed between calibration and validation. 4. Although not defined for this purpose, our study provided alarming evidence regarding the impact of cargo‐ship traffic on invertebrate communities in river reaches. Reaches with cargo‐ship traffic were found to have more genera with long life cycles that reproduce repeatedly by ovoviviparity and have a sessile life. 5. The performance of our trait‐based approach to correctly assign reaches to either least impacted or impacted conditions should promote further research on the topic across larger geographic areas (without regionalization) and across smaller stream types to provide a powerful biomonitoring tool that fulfils current European Union directives.     

Relating dispersal and range expansion of California sea otters

Linking dispersal and range expansion of invasive species has long challenged theoretical and quantitative ecologists. Subtle differences in dispersal can yield large differences in geographic spread, with speeds ranging from constant to rapidly increasing. We developed a stage-structured integrodifference equation (IDE) model of the California sea otter range expansion that occurred between 1914 and 1986. The non-spatial model, a linear matrix population model, was coupled to a suite of candidate dispersal kernels to form stage-structured IDEs. Demographic and dispersal parameters were estimated independent of range expansion data. Using a single dispersal parameter, alpha, we examined how well these stage-structured IDEs related small scale demographic and dispersal processes with geographic population expansion. The parameter alpha was estimated by fitting the kernels to dispersal data and by fitting the IDE model to range expansion data. For all kernels, the alpha estimate from range expansion data fell within the 95% confidence intervals of the alpha estimate from dispersal data. The IDE models with exponentially bounded kernels predicted invasion velocities that were captured within the 95% confidence bounds on the observed northbound invasion velocity. However, the exponentially bounded kernels yielded range expansions that were in poor qualitative agreement with range expansion data. An IDE model with fat (exponentially unbounded) tails and accelerating spatial spread yielded the best qualitative match. This model explained 94% and 97% of the variation in northbound and southbound range expansions when fit to range expansion data. These otters may have been fat-tailed accelerating invaders or they may have followed a piece-wise linear spread first over kelp forests and then over sandy habitats. Further, habitat-specific dispersal data could resolve these explanations.     

EFOMP policy statement 16: The role and competences of medical physicists and medical physics experts under 2013/59/EURATOM

On 5 December 2013 the European Council promulgated Directive 2013/59/EURATOM. This Directive is important for Medical Physicists and Medical Physics Experts as it puts the profession on solid foundations and describes it more comprehensively. Much commentary regarding the role and competences has been developed in the context of the European Commission project “European Guidelines on the Medical Physics Expert” published as Radiation Protection Report RP174. The guidelines elaborate on the role and responsibilities under 2013/59/EURATOM in terms of a mission statement and competence profile in the specialty areas of Medical Physics relating to medical radiological services, namely Diagnostic and Interventional Radiology, Radiation Oncology and Nuclear Medicine. The present policy statement summarises the provisions of Directive 2013/59/EURATOM regarding the role and competences, reiterates the results of the European Guidelines on the Medical Physics Expert document relating to role and competences of the profession and provides additional commentary regarding further issues arising following the publication of the RP174 guidelines.     

Interlaboratory comparison of the dicentric chromosome assay for radiation biodosimetry in mass casualty events

This interlaboratory comparison validates the dicentric chromosome assay for assessing radiation dose in mass casualty accidents and identifies the advantages and limitations of an international biodosimetry network. The assay's validity and accuracy were determined among five laboratories following the International Organization for Standardization guidelines. Blood samples irradiated at the Armed Forces Radiobiology Research Institute were shipped to all laboratories, which constructed individual radiation calibration curves and assessed the dose to dose-blinded samples. Each laboratory constructed a dose-effect calibration curve for the yield of dicentrics for (60)Co gamma rays in the 0 to 5-Gy range, using the maximum likelihood linear-quadratic model, Y = c + alphaD + betaD(2). For all laboratories, the estimated coefficients of the fitted curves were within the 99.7% confidence intervals (CIs), but the observed dicentric yields differed. When each laboratory assessed radiation doses to four dose-blinded blood samples by comparing the observed dicentric yield with the laboratory's own calibration curve, the estimates were accurate in all laboratories at all doses. For all laboratories, actual doses were within the 99.75% CI for the assessed dose. Across the dose range, the error in the estimated doses, compared to the physical doses, ranged from 15% underestimation to 15% overestimation.     

Classification of ECG arrhythmia by a modular neural network based on Mixture of Experts and Negatively Correlated Learning

In this paper, we propose a novel ECG arrhythmia classification method using the complementary features of Mixture of Experts (ME) and Negatively Correlated Learning (NCL). Negative Correlation Learning and Mixture of Experts methods utilize different error functions for simultaneous training of negatively correlated Neural Networks. The capability of a control parameter for NCL is incorporated in the error function of ME, which enables the training algorithm of ME to establish a balance in bias-variance-covariance trade-offs. ECG records from the MIT-BIH arrhythmia database are selected as test data. It is observed that the proposed classification method not only preserves the advantages and alleviates the disadvantages of its basis approaches, but also offering significantly improved performance over the original methods.