Evaluating the accuracy and calibration of expert predictions under uncertainty: predicting the outcomes of ecological research

Aim Expert knowledge routinely informs ecological research and decision‐making. Its reliability is often questioned, but is rarely subject to empirical testing and validation. We investigate the ability of experts to make quantitative predictions of variables for which the answers are known. Location Global. Methods Experts in four ecological subfields were asked to make predictions about the outcomes of scientific studies, in the form of unpublished (in press) journal articles, based on information in the article introduction and methods sections. Estimates from students were elicited for one case study for comparison. For each variable, participants assessed a lower and upper bound, best guess and level of confidence that the observed value will lie within their ascribed interval. Responses were assessed for (1) accuracy: the degree to which predictions corresponded with observed experimental results, (2) informativeness: precision of the uncertainty bounds, and (3) calibration: degree to which the uncertainty bounds contained the truth as often as specified. Results Expert responses were found to be overconfident, specifying 80% confidence intervals that captured the truth only 49–65% of the time. In contrast, student 80% intervals captured the truth 76% of the time, displaying close to perfect calibration. Best estimates from experts were on average more accurate than those from students. The best students outperformed the worst experts. No consistent relationships were observed between performance and years of experience, publication record or self‐assessment of expertise. Main conclusions Experts possess valuable knowledge but may require training to communicate this knowledge accurately. Expert status is a poor guide to good performance. In the absence of training and information on past performance, simple averages of expert responses provide a robust counter to individual variation in performance.     

A Smartphone App and Cloud-Based Consultation System for Burn Injury Emergency Care

Background

Each year more than 10 million people worldwide are burned severely enough to require medical attention, with clinical outcomes noticeably worse in resource poor settings. Expert clinical advice on acute injuries can play a determinant role and there is a need for novel approaches that allow for timely access to advice. We developed an interactive mobile phone application that enables transfer of both patient data and pictures of a wound from the point-of-care to a remote burns expert who, in turn, provides advice back.

Methods and Results

The application is an integrated clinical decision support system that includes a mobile phone application and server software running in a cloud environment. The client application is installed on a smartphone and structured patient data and photographs can be captured in a protocol driven manner. The user can indicate the specific injured body surface(s) through a touchscreen interface and an integrated calculator estimates the total body surface area that the burn injury affects. Predefined standardised care advice including total fluid requirement is provided immediately by the software and the case data are relayed to a cloud server. A text message is automatically sent to a burn expert on call who then can access the cloud server with the smartphone app or a web browser, review the case and pictures, and respond with both structured and personalized advice to the health care professional at the point-of-care.

Conclusions

In this article, we present the design of the smartphone and the server application alongside the type of structured patient data collected together with the pictures taken at point-of-care. We report on how the application will be introduced at point-of-care and how its clinical impact will be evaluated prior to roll out. Challenges, strengths and limitations of the system are identified that may help materialising or hinder the expected outcome to provide a solution for remote consultation on burns that can be integrated into routine acute clinical care and thereby promote equity in injury emergency care, a growing public health burden.     

Learn from the Best

What is more inspiring than a discussion with the leading scientists in your field? As a student or a young researcher, you have likely been influenced by mentors guiding you in your career and leading you to your current position. Any discussion with or advice from an expert is certainly very helpful for young people. But how often do we have the opportunity to meet experts? Do we make the most out of these situations? Meetings organized for young scientists are a great opportunity not only for the attendees: they are an opportunity for experts to meet bright students and learn from them in return. In this article, we introduce several successful events organized by Regional Student Groups all around the world, bridging the gap between experts and young scientists. We highlight how rewarding it is for all participants: young researchers, experts, and organizers. We then discuss the various benefits and emphasize the importance of organizing and attending such meetings. As a young researcher, seeking mentorship and additional skills training is a crucial step in career development. Keep in mind that one day, you may be an inspiring mentor, too.     

The Nature of Expertise in Fingerprint Matching: Experts Can Do a Lot with a Little

Expert decision making often seems impressive, even miraculous. People with genuine expertise in a particular domain can perform quickly and accurately, and with little information. In the series of experiments presented here, we manipulate the amount of “information” available to a group of experts whose job it is to identify the source of crime scene fingerprints. In Experiment 1, we reduced the amount of information available to experts by inverting fingerprint pairs and adding visual noise. There was no evidence for an inversion effect—experts were just as accurate for inverted prints as they were for upright prints—but expert performance with artificially noisy prints was impressive. In Experiment 2, we separated matching and nonmatching print pairs in time. Experts were conservative, but they were still able to discriminate pairs of fingerprints that were separated by five-seconds, even though the task was quite different from their everyday experience. In Experiment 3, we separated the print pairs further in time to test the long-term memory of experts compared to novices. Long-term recognition memory for experts and novices was the same, with both performing around chance. In Experiment 4, we presented pairs of fingerprints quickly to experts and novices in a matching task. Experts were more accurate than novices, particularly for similar nonmatching pairs, and experts were generally more accurate when they had more time. It is clear that experts can match prints accurately when there is reduced visual information, reduced opportunity for direct comparison, and reduced time to engage in deliberate reasoning. These findings suggest that non-analytic processing accounts for a substantial portion of the variance in expert fingerprint matching accuracy. Our conclusion is at odds with general wisdom in fingerprint identification practice and formal training, and at odds with the claims and explanations that are offered in court during expert testimony.     

What is an expert? A systems perspective on expertise

Expert knowledge is a valuable source of information with a wide range of research applications. Despite the recent advances in defining expert knowledge, little attention has been given to how to view expertise as a system of interacting contributory factors for quantifying an individual's expertise. We present a systems approach to expertise that accounts for many contributing factors and their inter‐relationships and allows quantification of an individual's expertise. A Bayesian network (BN) was chosen for this purpose. For illustration, we focused on taxonomic expertise. The model structure was developed in consultation with taxonomists. The relative importance of the factors within the network was determined by a second set of taxonomists (supra‐experts) who also provided validation of the model structure. Model performance was assessed by applying the model to hypothetical career states of taxonomists designed to incorporate known differences in career states for model testing. The resulting BN model consisted of 18 primary nodes feeding through one to three higher‐order nodes before converging on the target node (Taxonomic Expert). There was strong consistency among node weights provided by the supra‐experts for some nodes, but not others. The higher‐order nodes, “Quality of work” and “Total productivity”, had the greatest weights. Sensitivity analysis indicated that although some factors had stronger influence in the outer nodes of the network, there was relatively equal influence of the factors leading directly into the target node. Despite the differences in the node weights provided by our supra‐experts, there was good agreement among assessments of our hypothetical experts that accurately reflected differences we had specified. This systems approach provides a way of assessing the overall level of expertise of individuals, accounting for multiple contributory factors, and their interactions. Our approach is adaptable to other situations where it is desirable to understand components of expertise.     

How Expert Advice Influences Decision Making

People often use expert advice when making decisions in our society, but how we are influenced by this advice has yet to be understood. To address this, using functional magnetic resonance imaging, we provided expert and novice advice to participants during an estimation task. Participants reported that they valued expert advice more than novice advice, and activity in the ventral striatum correlated with this valuation, even before decisions with the advice were made. When using advice, participants compared their initial opinion to their advisor’s opinion. This comparison, termed the “opinion difference”, influenced advice utilization and was represented in reward-sensitive brain regions. Finally, the left lateral orbitofrontal cortex integrated both the size of the opinion difference and the advisor’s level of expertise, and average activity in this area correlated with mean advice utilization across participants. Taken together, these findings provide neural evidence for how advice engenders behavioral change during the decision-making process.     

To model or not to model?

In theory, the combination of mathematical modeling with experimental studies can be a powerful and compelling approach to understanding cell biology. In practice, choosing appropriate problems, identifying willing and able collaborators, and publishing the resulting research can be remarkably challenging. To provide perspective on the question of whether and when to combine modeling and experiments, a panel of experts at the 2010 ASCB Annual Meeting shared their personal experiences and advice on how to use modeling effectively.     

How to succeed in science: a concise guide for young biomedical scientists. Part I: taking the plunge

Biomedical research has never been more intellectually exciting or practically important to society. Ironically, pursuing a career as a biomedical scientist has never been more difficult. Here I provide unvarnished advice for young biomedical scientists on the difficulties that lie ahead and on how to find the right laboratories for training in the skills that you will need to succeed. Although my advice is geared towards succeeding in the United States, many aspects apply to other countries.     

Environmentally Conscious Product Design: A Collaborative Internet-based Modeling Approach

This paper proposes a computer-based method for providing product designers with real-time environmental impact assessment. In this concurrent modeling approach, environmental experts build life-cycle models, define their interfaces, and publish them as distributed objects on the Internet. Traditional designers integrating these objects into their design models have access to the impact assessment methods provided by the environmental expert. In this paradigm, the focus shifts from providing techniques that let non-expert designers perform life-cycle impact assessments to tools that facilitate timely communication and information transfer between designers and appropriate environmental experts. Establishing real-time communication between the product design models and the environmental life-cycle models is the primary focus of this paper. Methods for establishing and maintaining the interaction between life-cycle and product design models are described. A beverage container design example illustrates how this collaborative approach can use environmental and traditional design goals to determine effective tradeoffs between design alternatives.     

Chapter 14: Cancer Genome Analysis

Although there is great promise in the benefits to be obtained by analyzing cancer genomes, numerous challenges hinder different stages of the process, from the problem of sample preparation and the validation of the experimental techniques, to the interpretation of the results. This chapter specifically focuses on the technical issues associated with the bioinformatics analysis of cancer genome data. The main issues addressed are the use of database and software resources, the use of analysis workflows and the presentation of clinically relevant action items. We attempt to aid new developers in the field by describing the different stages of analysis and discussing current approaches, as well as by providing practical advice on how to access and use resources, and how to implement recommendations. Real cases from cancer genome projects are used as examples.

What to Learn in This Chapter

This chapter presents an overview of how cancer genomes can be analyzed, discussing some of the challenges involved and providing practical advice on how to address them. As the primary analysis of experimental data is described elsewhere (sequencing, alignment and variant calling), we will focus on the secondary analysis of the data, i.e., the selection of candidate driver genes, functional interpretation and the presentation of the results. Emphasis is placed on how to build applications that meet the needs of researchers, academics and clinicians. The general features of such applications are laid out, along with advice on their design and implementation. This document should serve as a starter guide for bioinformaticians interested in the analysis of cancer genomes, although we also hope that more experienced bioinformaticians will find interesting solutions to some key technical issues.

This article is part of the “Translational Bioinformatics” collection for PLOS Computational Biology.

     

Experience Does Not Equal Expertise in Recognizing Infrequent Incoming Gunfire: Neural Markers for Experience and Task Expertise at Peak Behavioral Performance

For a soldier, decisions to use force can happen rapidly and sometimes lead to undesired consequences. In many of these situations, there is a rapid assessment by the shooter that recognizes a threat and responds to it with return fire. But the neural processes underlying these rapid decisions are largely unknown, especially amongst those with extensive weapons experience and expertise. In this paper, we investigate differences in weapons experts and non-experts during an incoming gunfire detection task. Specifically, we analyzed the electroencephalography (EEG) of eleven expert marksmen/soldiers and eleven non-experts while they listened to an audio scene consisting of a sequence of incoming and non-incoming gunfire events. Subjects were tasked with identifying each event as quickly as possible and committing their choice via a motor response. Contrary to our hypothesis, experts did not have significantly better behavioral performance or faster response time than novices. Rather, novices indicated trends of better behavioral performance than experts. These group differences were more dramatic in the EEG correlates of incoming gunfire detection. Using machine learning, we found condition-discriminating EEG activity among novices showing greater magnitude and covering longer periods than those found in experts. We also compared group-level source reconstruction on the maximum discriminating neural correlates and found that each group uses different neural structures to perform the task. From condition-discriminating EEG and source localization, we found that experts perceive more categorical overlap between incoming and non-incoming gunfire. Consequently, the experts did not perform as well behaviorally as the novices. We explain these unexpected group differences as a consequence of experience with gunfire not being equivalent to expertise in recognizing incoming gunfire.     

Evaluation of cell substrates for the production of biologicals: Revision of WHO recommendations: Report of the WHO Study Group on Cell Substrates for the Production of Biologicals, 22–23 April 2009, Bethesda,USA

Evaluating cell substrates for producing vaccines and other biologicals is one of the critical aspects in assuring quality and safety of these products. As part of its mission in setting standards for biological products, WHO provides recommendations for manufacturing and evaluating biologicals. Regular updates of the guidance documents are important to manufacturers and regulators worldwide. WHO Expert Committee on Biological Standardization (ECBS) identified a need for revising the requirements for cell substrates (WHO TRS 878, annex 1). In response, WHO established a Study Group (SG) in 2006 that prepared an updated set of recommendations for using cell substrates for the production of biologicals. A summary of the proposed changes that the SG made in 2007 is available at WHO web site (http://www.who.int/biologicals/publications/meetings/areas/vaccines/cells/en/index.html). Draft revised recommendations were circulated to regulators, manufacturers and other experts for comments in April 2009.The SG held its third meeting on 22–23 April 2009 to review progress in the revision and to propose further improvements. In addition, the experts discussed the need for reference preparations, reference cell banks, and standardization of testing methodologies. The SG proposed clarifications of the rationale for in vivo testing as well as the potential for applying new methods for in vitro testing for detecting microbial agents. In line with this, WHO should conduct review of the current manufacturers' practice in using tests for microbial agents and interpreting these results. Additionally, WHO should take a lead in developing an International Standard for nucleic acid amplification test (NAT) for detecting mycoplasma contamination in cell substrates. WHO Collaborating Centers will lead this initiative, involving other relevant institutions in this area. Finally, advice on the replacement of the WHO Vero reference cell bank 10–87 with respect to the source of cells and re-characterization of the bank was provided. The intended use of the replacement cell bank would be the same as for the current cell bank, which is to serve as a source of well-characterized cells for establishing master cell banks for the production of biologicals. The SG will report outcomes of its discussion to the ECBS at its next meeting in October 2009 for further considerations and advice regarding the proposed course of action.     

Do we know enough? A scientific and ethical analysis of the basis for genetic-based personalized nutrition

This article discusses the prospects and limitations of the scientific basis for offering personalized nutrition advice based upon individual genetic information. Two divergent scientific positions are presented, with an ethical comment. The crucial question is whether the current knowledge base is sufficiently strong for taking an ethically responsible decision to offer personalized nutrition advice based upon gene–diet–health interaction. According to the first position, the evidence base for translating the outcomes of nutrigenomics research into personalized nutritional advice is as yet immature. There is also limited evidence that genotype-based dietary advice will motivate appropriate behavior changes. Filling the gaps in our knowledge will require larger and better randomized controlled trials. According to the second position, personalized nutrition must be evaluated in relation to generally accepted standard dietary advice—partly derived from epidemiological observations and usually not proven by clinical trials. With personalized nutrition, we cannot demand stronger evidence. In several specific cases of gene–diet interaction, it may be more beneficial for individuals with specific genotypes to follow personalized advice rather than general dietary recommendations. The ethical comment, finally, considers the ethical aspects of deciding how to proceed in the face of such uncertainty. Two approaches for an ethically responsible way forward are proposed. Arguing from a precautionary approach, it is suggested that personalized dietary advice should be offered only when there is strong scientific evidence for health effects, followed by stepwise evaluation of unforeseen behavioral and psychological effects. Arguing from theoretical and applied ethics as well as psychology, it is also suggested that personalized advice should avoid paternalism and instead focus on supporting the autonomous choice of each person.     

Essential diagnostics--the role of the Department of Biomedical Research of the Royal Tropical Institute in the 21st century

In the light of emerging and overlooked infectious diseases and widespread drug resistance, diagnostics have become increasingly important in supporting surveillance, disease control and outbreak management programs. In many low-income countries the diagnostic service has been a neglected part of health care, often lacking quantity and quality or even non-existing at all. High-income countries have exploited few of their advanced technical abilities for the much-needed development of low-cost, rapid diagnostic tests to improve the accuracy of diagnosis and accelerate the start of appropriate treatment. As is now also recognized by World Health Organization, investment in the development of affordable diagnostic tools is urgently needed to further our ability to control a variety of diseases that form a major threat to humanity. The Royal Tropical Institute's Department of Biomedical Research aims to contribute to the health of people living in the tropics. To this end, its multidisciplinary group of experts focuses on the diagnosis of diseases that are major health problems in low-income countries. In partnership we develop, improve and evaluate simple and cheap diagnostic tests, and perform epidemiological studies. Moreover, we advice and support others--especially those in developing countries--in their efforts to diagnose infectious diseases.     

A Practical Guide for Improving Transparency and Reproducibility in Neuroimaging Research

Recent years have seen an increase in alarming signals regarding the lack of replicability in neuroscience, psychology, and other related fields. To avoid a widespread crisis in neuroimaging research and consequent loss of credibility in the public eye, we need to improve how we do science. This article aims to be a practical guide for researchers at any stage of their careers that will help them make their research more reproducible and transparent while minimizing the additional effort that this might require. The guide covers three major topics in open science (data, code, and publications) and offers practical advice as well as highlighting advantages of adopting more open research practices that go beyond improved transparency and reproducibility.     

Guidelines on appropriate indications for upper gastrointestinal endoscopy. Working Party of the Joint Committee of the Royal College of Physicians of London,Royal College of Surgeons of England,Royal College of Anaesthetists,Association of Surgeons,the British Society of Gastroenterology,and the Thoracic Society of Great Britain.

Upper gastrointestinal endoscopy is a valuable diagnostic tool, but for an endoscopy service to be effective it is essential that it is not overloaded with inappropriately referred patients. A joint working party in Britain has considered the available literature on indications for endoscopy, assessed standard practice through a questionnaire, and audited randomly selected cases using an independent panel of experts and an American database system. They used these data to produce guidelines on the appropriate and inappropriate indications for referral for endoscopy, although they emphasise that under certain circumstances there may be reasons to deviate from the advice given. The need for endoscopy is most difficult to judge in patients with dyspepsia, and this aspect is discussed in detail. Early endoscopy will often prove more cost effective than delaying until the indications are clearer.     

Experts and Novices Use the Same Factors–But Differently–To Evaluate Pearl Quality

Well-trained experts in pearl grading have been thought to evaluate pearls according to their glossiness, interference color, and shape. However, the characteristics of their evaluations are not fully understood. Using pearl grading experiments, we investigate the consistency of novice (i.e., without knowledge of pearl grading) and expert participants’ pearl grading skill and then compare the novices’ grading with that of experts; furthermore, we discuss the relationship between grading, interference color, and glossiness. We found that novices’ grading was significantly less concordant with experts average grading than was experts’ grading; more than half of novices graded pearls the opposite of how experts graded those same pearls. However, while experts graded pearls more consistently than novices did, novices’ consistency was relatively high. We also found differences between the groups in regression analyses that used interference color and glossiness as explanatory variables and were conducted for each trial. Although the regression coefficient was significant in 60% of novices’ trials, there were fewer significant trials for the experts (20%). This indicates that novices can also make use of these two factors, but that their usage is simpler than that of the experts. These results suggest that experts and novices share some values about pearls but that the evaluation method is elaborated for experts.     

Thriving in Multidisciplinary Research: Advice for New Bioinformatics Students

The sciences have seen a large increase in demand for students in bioinformatics and multidisciplinary fields in general. Many new educational programs have been created to satisfy this demand, but navigating these programs requires a non-traditional outlook and emphasizes working in teams of individuals with distinct yet complementary skill sets. Written from the perspective of a current bioinformatics student, this article seeks to offer advice to prospective and current students in bioinformatics regarding what to expect in their educational program, how multidisciplinary fields differ from more traditional paths, and decisions that they will face on the road to becoming successful, productive bioinformaticists.     

It takes two-skilled recognition of objects engages lateral areas in both hemispheres

Our object recognition abilities, a direct product of our experience with objects, are fine-tuned to perfection. Left temporal and lateral areas along the dorsal, action related stream, as well as left infero-temporal areas along the ventral, object related stream are engaged in object recognition. Here we show that expertise modulates the activity of dorsal areas in the recognition of man-made objects with clearly specified functions. Expert chess players were faster than chess novices in identifying chess objects and their functional relations. Experts' advantage was domain-specific as there were no differences between groups in a control task featuring geometrical shapes. The pattern of eye movements supported the notion that experts' extensive knowledge about domain objects and their functions enabled superior recognition even when experts were not directly fixating the objects of interest. Functional magnetic resonance imaging (fMRI) related exclusively the areas along the dorsal stream to chess specific object recognition. Besides the commonly involved left temporal and parietal lateral brain areas, we found that only in experts homologous areas on the right hemisphere were also engaged in chess specific object recognition. Based on these results, we discuss whether skilled object recognition does not only involve a more efficient version of the processes found in non-skilled recognition, but also qualitatively different cognitive processes which engage additional brain areas.