Assessing Discriminative Performance at External Validation of Clinical Prediction Models

Introduction

External validation studies are essential to study the generalizability of prediction models. Recently a permutation test, focusing on discrimination as quantified by the c-statistic, was proposed to judge whether a prediction model is transportable to a new setting. We aimed to evaluate this test and compare it to previously proposed procedures to judge any changes in c-statistic from development to external validation setting.

Methods

We compared the use of the permutation test to the use of benchmark values of the c-statistic following from a previously proposed framework to judge transportability of a prediction model. In a simulation study we developed a prediction model with logistic regression on a development set and validated them in the validation set. We concentrated on two scenarios: 1) the case-mix was more heterogeneous and predictor effects were weaker in the validation set compared to the development set, and 2) the case-mix was less heterogeneous in the validation set and predictor effects were identical in the validation and development set. Furthermore we illustrated the methods in a case study using 15 datasets of patients suffering from traumatic brain injury.

Results

The permutation test indicated that the validation and development set were homogenous in scenario 1 (in almost all simulated samples) and heterogeneous in scenario 2 (in 17%-39% of simulated samples). Previously proposed benchmark values of the c-statistic and the standard deviation of the linear predictors correctly pointed at the more heterogeneous case-mix in scenario 1 and the less heterogeneous case-mix in scenario 2.

Conclusion

The recently proposed permutation test may provide misleading results when externally validating prediction models in the presence of case-mix differences between the development and validation population. To correctly interpret the c-statistic found at external validation it is crucial to disentangle case-mix differences from incorrect regression coefficients.     

External Validation of Models for Prediction of Lymph Node Metastasis in Urothelial Carcinoma of the Bladder

Purpose

To externally validate models to predict LN metastsis; Karakiewicz nomogram, clinical nodal staging score (cNSS), and pathologic nodal staging score (pNSS) using a different cohort

Materials and Methods

Clinicopathologic data from 500 patients who underwent radical cystectomy and pelvic lymphadenectomy were analyzed. The overall predictive values of models were compared with the criteria of overall performance, discrimination, calibration, and clinical usefulness.

Results

Presence of pN+ stages was recorded in 117 patients (23.4%). Agreement between clinical and pathologic stage was noted in 174 (34.8%). Based on Nagelkerke’s peudo-R² and brier score, pNSS demonstrated best overall performance. Area under the receiver operating characteristics curve, showed that pNSS had the best discriminatory ability. In all models, calibration was on average correct (calibration-in-the-large coefficient = zero). On decision curve analysis, pNSS performed better than other models across a wide range of threshold probabilities.

Conclusions

When compared to pNSS, current precystectomy models such as the Karakiewicz nomogram and cNSS cannot predict the probability of LN metastases accurately. The findings suggest that the application of pNSS to Asian patients is feasible.     

高危结直肠腺瘤的危险因素及风险预测模型的构建与验证

摘要 目的：探讨高危结直肠腺瘤的影响因素,构建风险预测模型并验证。方法：回顾性分析2021年1月至2021年12月期间在江苏大学附属人民医院进行诊疗的1408例结直肠腺瘤患者的资料,根据病理特征分为高危结直肠腺瘤组（759例）和非高危结直肠腺瘤组（649例）。采用Logistic回归分析筛选高危结直肠腺瘤的独立危险因素并建立风险预测模型,并验证预测模型的应用效能。结果：Logistic回归分析结果显示,病灶部位为直肠、高血压、高脂血症、年龄≥53岁、吸烟是高危结直肠腺瘤的独立危险因素（P＜0.05）。基于以上因素建立预测高危结直肠腺瘤风险的列线图模型,经Hosmer-Lemeshow检验和受试者工作特征曲线（ROC）分析显示,该风险预测模型具有较好的拟合度和预测效能,可以用于高危腺瘤的风险预测。结论：病灶部位为直肠、高血压、高脂血症、年龄≥53岁、吸烟是高危结直肠腺瘤的独立危险因素,临床医生可尽早对高危患者进行预防性干预以减缓高危腺瘤的发生。     

A Model-Based Approach to Assess Epidemic Risk

Statistics in Biosciences - We study how international flights can facilitate the spread of an epidemic to a worldwide scale. We combine an infrastructure network of flight connections with a...     

External Validation of Prediction Models for Pneumonia in Primary Care Patients with Lower Respiratory Tract Infection: An Individual Patient Data Meta-Analysis

Background

Pneumonia remains difficult to diagnose in primary care. Prediction models based on signs and symptoms (S&S) serve to minimize the diagnostic uncertainty. External validation of these models is essential before implementation into routine practice. In this study all published S&S models for prediction of pneumonia in primary care were externally validated in the individual patient data (IPD) of previously performed diagnostic studies.

Methods and Findings

S&S models for diagnosing pneumonia in adults presenting to primary care with lower respiratory tract infection and IPD for validation were identified through a systematical search. Six prediction models and IPD of eight diagnostic studies (N total = 5308, prevalence pneumonia 12%) were included. Models were assessed on discrimination and calibration. Discrimination was measured using the pooled Area Under the Curve (AUC) and delta AUC, representing the performance of an individual model relative to the average dataset performance. Prediction models by van Vugt et al. and Heckerling et al. demonstrated the highest pooled AUC of 0.79 (95% CI 0.74–0.85) and 0.72 (0.68–0.76), respectively. Other models by Diehr et al., Singal et al., Melbye et al., and Hopstaken et al. demonstrated pooled AUCs of 0.65 (0.61–0.68), 0.64 (0.61–0.67), 0.56 (0.49–0.63) and 0.53 (0.5–0.56), respectively. A similar ranking was present based on the delta AUCs of the models. Calibration demonstrated close agreement of observed and predicted probabilities in the models by van Vugt et al. and Singal et al., other models lacked such correspondence. The absence of predictors in the IPD on dataset level hampered a systematical comparison of model performance and could be a limitation to the study.

Conclusions

The model by van Vugt et al. demonstrated the highest discriminative accuracy coupled with reasonable to good calibration across the IPD of different study populations. This model is therefore the main candidate for primary care use.     

External Validation of a Predictive Model for Acute Pancreatitis Risk in Patients with Severe Hypertriglyceridemia

Objective: We previously developed a predictive model to assess the risk of developing acute pancreatitis (AP) in patients with severe hypertriglyceridemia (HTG). In this study, we aimed to externally validate this model.Methods: The validation cohort included cross-sectional data between 2013 and 2017. Adult patients (≥18 years old) with triglyceride levels ≥1,000 mg/dL were identified. Based on our previous 4-factor predictive model (age, triglyceride &lsqb;TG], excessive alcohol use, and gallstone disease), we estimated the probability of developing AP. Model performance was assessed using area under receiver operating characteristic curve (AUROC).Results: In comparison to the original cohort, patients in the validation cohort had more prevalent acute pancreatitis (16.2% versus 9.2%; P<.001) and gallstone disease (7.5% versus 2.1%; P<.001). Other characteristics were comparable and not statistically significant. The AUROCs were almost identical: 0.8337 versus 0.8336 in the validation and the original cohorts, respectively. In univariable analyses, the highest increase in odds of AP was associated with HTG, followed by gallstones, excessive alcohol use, and younger age.Conclusion: This study externally validates the 4-factor predictive model to estimate the risk of AP in adult patients with severe HTG (TG ≥1,000 mg/dL). Younger age was confirmed to place patients at high risk of AP. The clinical risk categories suggested in this study may be useful to guide treatment options.Abbreviations: AP = acute pancreatitis; ASCVD = atherosclerotic cardiovascular disease; AUROC = area under the receiver operating characteristic curve; FRAX = fracture risk assessment tool; HTG = hypertriglyceridemia; OR = odds ratio; TG = triglyceride level     

A Weighted Sample Framework to Incorporate External Calculators for Risk Modeling

Statistics in Biosciences - Personalized risk prediction calculators abound in medicine, and they carry important information about the effect of prognostic factors on outcomes of interest. How to...     

Extending Tests of Random Effects to Assess for Measurement Invariance in Factor Models

Factor analysis models are widely used in health research to summarize hard-to-measure predictor or outcome variable constructs. For example, in the ELEMENT study, factor models are used to summarize lead exposure biomarkers which are thought to indirectly measure prenatal exposure to lead. Classic latent factor models are fitted assuming that factor loadings are constant across all covariate levels (e.g., maternal age in ELEMENT); that is, measurement invariance (MI) is assumed. When the MI is not met, measurement bias is introduced. Traditionally, MI is examined by defining subgroups of the data based on covariates, fitting multi-group factor analysis, and testing differences in factor loadings across covariate groups. In this paper, we develop novel tests of measurement invariance by modeling the factor loadings as varying coefficients, i.e., letting the factor loading vary across continuous covariate values instead of groups. These varying coefficients are estimated using penalized splines, where spline coefficients are penalized by treating them as random coefficients. The test of MI is then carried out by conducting a likelihood ratio test for the null hypothesis that the variance of the random spline coefficients equals zero. We use a Monte Carlo EM algorithm for estimation, and obtain the likelihood using Monte Carlo integration. Using simulations, we compare the Type I error and power of our testing approach and the multi-group testing method. We apply the proposed methods to summarize data on prenatal biomarkers of lead exposure from the ELEMENT study and find violations of MI due to maternal age.     

A Real-Time Impedance Based Method to Assess Rhodococcus equi Virulence

Rhodococcus equi is a facultative intracellular pathogen of macrophages and the causative agent of foal pneumonia. R. equi virulence is usually assessed by analyzing intracellular growth in macrophages by enumeration of bacteria following cell lysis, which is time consuming and does not allow for a high throughput analysis. This paper describes the use of an impedance based real-time method to characterize proliferation of R. equi in macrophages, using virulent and attenuated strains lacking the vapA gene or virulence plasmid. Image analysis suggested that the time-dependent cell response profile (TCRP) is governed by cell size and roundness as well as cytoxicity of infecting R. equi strains. The amplitude and inflection point of the resulting TCRP were dependent on the multiplicity of infection as well as virulence of the infecting strain, thus distinguishing between virulent and attenuated strains.     

The Cultural Validation of Two Scales to Assess Social Stigma in Leprosy

Background

Stigma plays in an important role in the lives of persons affected by neglected tropical diseases, and assessment of stigma is important to document this. The aim of this study is to test the cross-cultural validity of the Community Stigma Scale (EMIC-CSS) and the Social Distance Scale (SDS) in the field of leprosy in Cirebon District, Indonesia.

Methodology/principle findings

Cultural equivalence was tested by assessing the conceptual, item, semantic, operational and measurement equivalence of these instruments. A qualitative exploratory study was conducted to increase our understanding of the concept of stigma in Cirebon District. A process of translation, discussions, trainings and a pilot study followed. A sample of 259 community members was selected through convenience sampling and 67 repeated measures were obtained to assess the psychometric measurement properties. The aspects and items in the SDS and EMIC-CSS seem equally relevant and important in the target culture. The response scales were adapted to ensure that meaning is transferred accurately and no changes to the scale format (e.g. lay out, statements or questions) of both scales were made. A positive correlation was found between the EMIC-CSS and the SDS total scores (r = 0.41). Cronbach''s alphas of 0.83 and 0.87 were found for the EMIC-CSS and SDS. The exploratory factor analysis indicated for both scales an adequate fit as unidimensional scale. A standard error of measurement of 2.38 was found in the EMIC-CSS and of 1.78 in the SDS. The test-retest reliability coefficient was respectively, 0.84 and 0.75. No floor or ceiling effects were found.

Conclusions/significance

According to current international standards, our findings indicate that the EMIC-CSS and the SDS have adequate cultural validity to assess social stigma in leprosy in the Bahasa Indonesia-speaking population of Cirebon District. We believe the scales can be further improved, for instance, by adding, changing and rephrasing certain items. Finally, we provide suggestions for use with other neglected tropical diseases.     

A Continental-Scale Validation of Ecosystem Service Models

Ecosystems - Faced with environmental degradation, governments worldwide are developing policies to safeguard ecosystem services (ES). Many ES models exist to support these policies, but they are...     

Validation of Ten Noninvasive Diagnostic Models for Prediction of Liver Fibrosis in Patients with Chronic Hepatitis B

Background and Aims

Noninvasive models have been developed for fibrosis assessment in patients with chronic hepatitis B. However, the sensitivity, specificity and diagnostic accuracy in evaluating liver fibrosis of these methods have not been validated and compared in the same group of patients. The aim of this study was to verify the diagnostic performance and reproducibility of ten reported noninvasive models in a large cohort of Asian CHB patients.

Methods

The diagnostic performance of ten noninvasive models (HALF index, FibroScan, S index, Zeng model, Youyi model, Hui model, APAG, APRI, FIB-4 and FibroTest) was assessed against the liver histology by ROC curve analysis in CHB patients. The reproducibility of the ten models were evaluated by recalculating the diagnostic values at the given cut-off values defined by the original studies.

Results

Six models (HALF index, FibroScan, Zeng model, Youyi model, S index and FibroTest) had AUROCs higher than 0.70 in predicting any fibrosis stage and 2 of them had best diagnostic performance with AUROCs to predict F≥2, F≥3 and F4 being 0.83, 0.89 and 0.89 for HALF index, 0.82, 0.87 and 0.87 for FibroScan, respectively. Four models (HALF index, FibroScan, Zeng model and Youyi model) showed good diagnostic values at given cut-offs.

Conclusions

HALF index, FibroScan, Zeng model, Youyi model, S index and FibroTest show a good diagnostic performance and all of them, except S index and FibroTest, have good reproducibility for evaluating liver fibrosis in CHB patients.

Registration Number

ChiCTR-DCS-07000039.     

Using Multiscale Spatial Models to Assess Potential Surrogate Habitat for an Imperiled Reptile

In evaluating conservation and management options for species, practitioners might consider surrogate habitats at multiple scales when estimating available habitat or modeling species’ potential distributions based on suitable habitats, especially when native environments are rare. Species’ dependence on surrogates likely increases as optimal habitat is degraded and lost due to anthropogenic landscape change, and thus surrogate habitats may be vital for an imperiled species’ survival in highly modified landscapes. We used spatial habitat models to examine a potential surrogate habitat for an imperiled ambush predator (eastern diamondback rattlesnake, Crotalus adamanteus; EDB) at two scales. The EDB is an apex predator indigenous to imperiled longleaf pine ecosystems (Pinus palustris) of the southeastern United States. Loss of native open-canopy pine savannas and woodlands has been suggested as the principal cause of the species’ extensive decline. We examined EDB habitat selection in the Coastal Plain tidewater region to evaluate the role of marsh as a potential surrogate habitat and to further quantify the species’ habitat requirements at two scales: home range (HR) and within the home range (WHR). We studied EDBs using radiotelemetry and employed an information-theoretic approach and logistic regression to model habitat selection as use vs. availability. We failed to detect a positive association with marsh as a surrogate habitat at the HR scale; rather, EDBs exhibited significantly negative associations with all landscape patches except pine savanna. Within home range selection was characterized by a negative association with forest and a positive association with ground cover, which suggests that EDBs may use surrogate habitats of similar structure, including marsh, within their home ranges. While our HR analysis did not support tidal marsh as a surrogate habitat, marsh may still provide resources for EDBs at smaller scales.     

Validation and Comparison of Two 30-Day Re-Admission Prediction Models in Patients with Diabetes

Objective: The objective was to evaluate the 30-day re-admission predictive performance of the HOSPITAL score and Diabetes Early Re-admission Risk Indicator (DERRI™) in hospitalized diabetes patients.Methods: This was a case-control study in an academic, tertiary center in the United States. Adult hospitalized diabetes patients were randomly identified between January 1, 2014, and September 30, 2017. Patients were categorized into two groups: (1) re-admitted within 30 days, and (2) not re-admitted within 30 days. Predictive performance of the HOSPITAL and DERRI™ scores was evaluated by calculating receiver operating characteristics curves (c-statistic), Hosmer-Lemeshow goodness-of-fit tests, and Brier scores.Results: A total of 200 patients were included (100 re-admitted, 100 non–re-admitted). The HOSPITAL score had a c-statistic of 0.731 (95% confidence interval &lsqb;CI], 0.661 to 0.800), Hosmer-Lemeshow test P = .211, and Brier score 0.212. The DERRI™ score had a c-statistic of 0.796 (95% CI, 0.734 to 0.857), Hosmer-Lemeshow test P = .114, and Brier score 0.212. The difference in receiver operating characteristic curves was not statistically significant between the two scores but showed a higher c-statistic with the DERRI™ score (P = .055).Conclusion: Both HOSPITAL and DERRI™ scores showed good predictive performance in 30-day re-admission of adult hospitalized diabetes patients. There was no significant difference in discrimination and calibration between the scores.Abbreviations: CI = confidence interval; DERRI™ = Diabetes Early Re-admission Risk Indicator; IQR = interquartile range     

Validation of the TracmorD Triaxial Accelerometer to Assess Physical Activity in Preschool Children

Objectives: To assess validity evidence of Tracmor_D to determine energy used for physical activity in 3‐4‐year‐old children. Design and Methods: Participants were randomly selected from GECKO Drenthe cohort (n = 30, age 3.4 ± 0.3 years). Total energy expenditure (TEE) was measured using the doubly labeled water method. Sleeping metabolic rate (SMR) was measured by indirect calorimetry (Deltatrac). TEE and SMR were used to calculate physical activity level (PAL) and activity energy expenditure (AEE). Physical activity was monitored using a DirectLife triaxial accelerometer, Tracmor_D with activity counts per minute (ACM) and activity counts per day (ACD) as outcome measures. Results: The best predictor for PAL was ACM with gender and weight, the best predictor for AEE was ACM alone (backward regression, R² = 0.50, P = 0.010 and R² = 0.31, P = 0.011, respectively). With ACD, the prediction model for PAL included ACD, height, gender, and sleep duration (R² = 0.48, P = 0.033), the prediction model for AEE included ACD, gender and sleep duration (R² = 0.39, P = 0.042). The accelerometer was worn for 5 days, but 3 days did not give a different estimated PAL. Conclusion: Tracmor_D provides moderate‐to‐strong validity evidence that supports its use to evaluate energy used for physical activity in 3‐4‐year‐old children.     

Development and Validation of Experimental Protocols for Use of Cardinal Models for Prediction of Microorganism Growth in Food Products

An experimental protocol to validate secondary-model application to foods was suggested. Escherichia coli, Listeria monocytogenes, Bacillus cereus, Clostridium perfringens, and Salmonella were observed in various food categories, such as meat, dairy, egg, or seafood products. The secondary model validated in this study was based on the gamma concept, in which the environmental factors temperature, pH, and water activity (aw) were introduced as individual terms with microbe-dependent parameters, and the effect of foodstuffs on the growth rates of these species was described with a food- and microbe-dependent parameter. This food-oriented approach was carried out by challenge testing, generally at 15 and 10°C for L. monocytogenes, E. coli, B. cereus, and Salmonella and at 25 and 20°C for C. perfringens. About 222 kinetics in foods were generated. The results were compared to simulations generated by existing software, such as PMP. The bias factor was also calculated. The methodology to obtain a food-dependent parameter (fitting step) and therefore to compare results given by models with new independent data (validation step) is discussed in regard to its food safety application. The proposed methods were used within the French national program of predictive microbiology, Sym′Previus, to include challenge test results in the database and to obtain predictive models designed for microbial growth in food products.     

A Fuzzy Logic Approach to Assess,Manage, and Communicate Carcinogenic Risk

A prospective approach to addressing carcinogen risk assessment is presented. Fuzzy reasoning is used to assess carcinogenic risk, characterize it, and control it. The approach is inspired by fuzzy control inference that deploys linguistic intelligence as input to a system described numerically through membership functions. Fuzzy-based reasoning to estimate carcinogenic risk provides several advantages as discussed here. The fuzzy reasoning approach has more capabilities than traditional models in dealing with risk agents that are probably carcinogens, possibly carcinogens, not classifiable as carcinogens, and probably not carcinogens. Input–output surfaces are presented for each hazard group to enable fast inferencing. Then, a hypothetical example is given to compare the results of traditional methods and the fuzzy-based approach to estimating the risk of a carcinogen to a human population. Results show similarity in risk characterization with less input information to the fuzzy-based approach. Fuzzy reasoning characterizes risk in more explicit and easy to grasp terms. Two outputs of the inferencing system are risk characterization and risk control or remediation.