Developing risk models for multicenter data using standard logistic regression produced suboptimal predictions: A simulation study

Although multicenter data are common, many prediction model studies ignore this during model development. The objective of this study is to evaluate the predictive performance of regression methods for developing clinical risk prediction models using multicenter data, and provide guidelines for practice. We compared the predictive performance of standard logistic regression, generalized estimating equations, random intercept logistic regression, and fixed effects logistic regression. First, we presented a case study on the diagnosis of ovarian cancer. Subsequently, a simulation study investigated the performance of the different models as a function of the amount of clustering, development sample size, distribution of center-specific intercepts, the presence of a center-predictor interaction, and the presence of a dependency between center effects and predictors. The results showed that when sample sizes were sufficiently large, conditional models yielded calibrated predictions, whereas marginal models yielded miscalibrated predictions. Small sample sizes led to overfitting and unreliable predictions. This miscalibration was worse with more heavily clustered data. Calibration of random intercept logistic regression was better than that of standard logistic regression even when center-specific intercepts were not normally distributed, a center-predictor interaction was present, center effects and predictors were dependent, or when the model was applied in a new center. Therefore, to make reliable predictions in a specific center, we recommend random intercept logistic regression.     

Comparison of complex modeling strategies for prediction of a binary outcome based on a few,highly correlated predictors

Motivated by a clinical prediction problem, a simulation study was performed to compare different approaches for building risk prediction models. Robust prediction models for hospital survival in patients with acute heart failure were to be derived from three highly correlated blood parameters measured up to four times, with predictive ability having explicit priority over interpretability. Methods that relied only on the original predictors were compared with methods using an expanded predictor space including transformations and interactions. Predictors were simulated as transformations and combinations of multivariate normal variables which were fitted to the partly skewed and bimodally distributed original data in such a way that the simulated data mimicked the original covariate structure. Different penalized versions of logistic regression as well as random forests and generalized additive models were investigated using classical logistic regression as a benchmark. Their performance was assessed based on measures of predictive accuracy, model discrimination, and model calibration. Three different scenarios using different subsets of the original data with different numbers of observations and events per variable were investigated. In the investigated setting, where a risk prediction model should be based on a small set of highly correlated and interconnected predictors, Elastic Net and also Ridge logistic regression showed good performance compared to their competitors, while other methods did not lead to substantial improvements or even performed worse than standard logistic regression. Our work demonstrates how simulation studies that mimic relevant features of a specific data set can support the choice of a good modeling strategy.     

Making predictions of mangrove deforestation: a comparison of two methods in Kenya

Deforestation of mangroves is of global concern given their importance for carbon storage, biogeochemical cycling and the provision of other ecosystem services, but the links between rates of loss and potential drivers or risk factors are rarely evaluated. Here, we identified key drivers of mangrove loss in Kenya and compared two different approaches to predicting risk. Risk factors tested included various possible predictors of anthropogenic deforestation, related to population, suitability for land use change and accessibility. Two approaches were taken to modelling risk; a quantitative statistical approach and a qualitative categorical ranking approach. A quantitative model linking rates of loss to risk factors was constructed based on generalized least squares regression and using mangrove loss data from 1992 to 2000. Population density, soil type and proximity to roads were the most important predictors. In order to validate this model it was used to generate a map of losses of Kenyan mangroves predicted to have occurred between 2000 and 2010. The qualitative categorical model was constructed using data from the same selection of variables, with the coincidence of different risk factors in particular mangrove areas used in an additive manner to create a relative risk index which was then mapped. Quantitative predictions of loss were significantly correlated with the actual loss of mangroves between 2000 and 2010 and the categorical risk index values were also highly correlated with the quantitative predictions. Hence, in this case the relatively simple categorical modelling approach was of similar predictive value to the more complex quantitative model of mangrove deforestation. The advantages and disadvantages of each approach are discussed, and the implications for mangroves are outlined.     

Selecting Optimal Random Forest Predictive Models: A Case Study on Predicting the Spatial Distribution of Seabed Hardness

Spatially continuous predictions of seabed hardness are important baseline environmental information for sustainable management of Australia’s marine jurisdiction. Seabed hardness is often inferred from multibeam backscatter data with unknown accuracy and can be inferred from underwater video footage at limited locations. In this study, we classified the seabed into four classes based on two new seabed hardness classification schemes (i.e., hard90 and hard70). We developed optimal predictive models to predict seabed hardness using random forest (RF) based on the point data of hardness classes and spatially continuous multibeam data. Five feature selection (FS) methods that are variable importance (VI), averaged variable importance (AVI), knowledge informed AVI (KIAVI), Boruta and regularized RF (RRF) were tested based on predictive accuracy. Effects of highly correlated, important and unimportant predictors on the accuracy of RF predictive models were examined. Finally, spatial predictions generated using the most accurate models were visually examined and analysed. This study confirmed that: 1) hard90 and hard70 are effective seabed hardness classification schemes; 2) seabed hardness of four classes can be predicted with a high degree of accuracy; 3) the typical approach used to pre-select predictive variables by excluding highly correlated variables needs to be re-examined; 4) the identification of the important and unimportant predictors provides useful guidelines for further improving predictive models; 5) FS methods select the most accurate predictive model(s) instead of the most parsimonious ones, and AVI and Boruta are recommended for future studies; and 6) RF is an effective modelling method with high predictive accuracy for multi-level categorical data and can be applied to ‘small p and large n’ problems in environmental sciences. Additionally, automated computational programs for AVI need to be developed to increase its computational efficiency and caution should be taken when applying filter FS methods in selecting predictive models.     

Enhancing long-term forecasting: Learning from COVID-19 models

While much effort has gone into building predictive models of the COVID-19 pandemic, some have argued that early exponential growth combined with the stochastic nature of epidemics make the long-term prediction of contagion trajectories impossible. We conduct two complementary studies to assess model features supporting better long-term predictions. First, we leverage the diverse models contributing to the CDC repository of COVID-19 USA death projections to identify factors associated with prediction accuracy across different projection horizons. We find that better long-term predictions correlate with: (1) capturing the physics of transmission (instead of using black-box models); (2) projecting human behavioral reactions to an evolving pandemic; and (3) resetting state variables to account for randomness not captured in the model before starting projection. Second, we introduce a very simple model, SEIRb, that incorporates these features, and few other nuances, offers informative predictions for as far as 20-weeks ahead, with accuracy comparable with the best models in the CDC set. Key to the long-term predictive power of multi-wave COVID-19 trajectories is capturing behavioral responses endogenously: balancing feedbacks where the perceived risk of death continuously changes transmission rates through the adoption and relaxation of various Non-Pharmaceutical Interventions (NPIs).     

A joint fairness model with applications to risk predictions for underrepresented populations

In data collection for predictive modeling, underrepresentation of certain groups, based on gender, race/ethnicity, or age, may yield less accurate predictions for these groups. Recently, this issue of fairness in predictions has attracted significant attention, as data-driven models are increasingly utilized to perform crucial decision-making tasks. Existing methods to achieve fairness in the machine learning literature typically build a single prediction model in a manner that encourages fair prediction performance for all groups. These approaches have two major limitations: (i) fairness is often achieved by compromising accuracy for some groups; (ii) the underlying relationship between dependent and independent variables may not be the same across groups. We propose a joint fairness model (JFM) approach for logistic regression models for binary outcomes that estimates group-specific classifiers using a joint modeling objective function that incorporates fairness criteria for prediction. We introduce an accelerated smoothing proximal gradient algorithm to solve the convex objective function, and present the key asymptotic properties of the JFM estimates. Through simulations, we demonstrate the efficacy of the JFM in achieving good prediction performance and across-group parity, in comparison with the single fairness model, group-separate model, and group-ignorant model, especially when the minority group's sample size is small. Finally, we demonstrate the utility of the JFM method in a real-world example to obtain fair risk predictions for underrepresented older patients diagnosed with coronavirus disease 2019 (COVID-19).     

Complexity is costly: a meta‐analysis of parametric and non‐parametric methods for short‐term population forecasting

Short‐term forecasts based on time series of counts or survey data are widely used in population biology to provide advice concerning the management, harvest and conservation of natural populations. A common approach to produce these forecasts uses time‐series models, of different types, fit to time series of counts. Similar time‐series models are used in many other disciplines, however relative to the data available in these other disciplines, population data are often unusually short and noisy and models that perform well for data from other disciplines may not be appropriate for population data. In order to study the performance of time‐series forecasting models for natural animal population data, we assembled 2379 time series of vertebrate population indices from actual surveys. Our data were comprised of three vastly different types: highly variable (marine fish productivity), strongly cyclic (adult salmon counts), and small variance but long‐memory (bird and mammal counts). We tested the predictive performance of 49 different forecasting models grouped into three broad classes: autoregressive time‐series models, non‐linear regression‐type models and non‐parametric time‐series models. Low‐dimensional parametric autoregressive models gave the most accurate forecasts across a wide range of taxa; the most accurate model was one that simply treated the most recent observation as the forecast. More complex parametric and non‐parametric models performed worse, except when applied to highly cyclic species. Across taxa, certain life history characteristics were correlated with lower forecast error; specifically, we found that better forecasts were correlated with attributes of slow growing species: large maximum age and size for fishes and high trophic level for birds. Synthesis Evaluating the data support for multiple plausible models has been an integral focus of many ecological analyses. However, the most commonly used tools to quantify support have weighted models’ hindcasting and forecasting abilities. For many applications, predicting the past may be of little interest. Concentrating only on the future predictive performance of time series models, we performed a forecasting competition among many different kinds of statistical models, applying each to many different kinds of vertebrate time series of population abundance. Low‐dimensional (simple) models performed well overall, but more complex models did slightly better when applied to time series of cyclic species (e.g. salmon).     

Modelling extinction risk in multispecies data sets: phylogenetically independent contrasts versus decision trees

Many recent studies of extinction risk have attempted to determine what differences exist between threatened and non-threatened species. One potential problem in such studies is that species-level data may contain phylogenetic non-independence. However, the use of phylogenetic comparative methods (PCM) to account for non-independence remains controversial, and some recent studies of extinction have recommended other methods that do not account for phylogenetic non-independence, notably decision trees (DTs). Here we perform a systematic comparison of techniques, comparing the performance of PCM regression models with corresponding non-phylogenetic regressions and DTs over different clades and response variables. We found that predictions were broadly consistent among techniques, but that predictive precision varied across techniques with PCM regression and DTs performing best. Additionally, despite their inability to account for phylogenetic non-independence, DTs were useful in highlighting interaction terms for inclusion in the PCM regression models. We discuss the implications of these findings for future comparative studies of extinction risk.     

基于树木起源、立地分级和龄组的单木生物量模型

以马尾松(Pinus massoniana)和落叶松(Larix)的大样本实测资料为建模样本,以独立抽取的样本为验证样本,把样本按起源、立地和龄组进行分级,采用与材积相容的两种相对生长方程,分普通最小二乘和两种加权最小二乘,对地上部分总生物量、地上各部分生物量和地下生物量进行模型拟合和验证,使用决定系数、均方根误差、总相对误差和估计精度等8项统计量对结果进行分析。结果表明:两个树种地上部分总生物量,立地分类方法,模型的拟合结果和适用性都最优;马尾松VAR模型较优,而落叶松CAR模型较好;两种加权最小二乘方法,在建模样本和验证样本中表现得不一致。在建模样本中,加权回归2(权重函数1/f^0.5)略优于加权回归1(权重函数1/y^0.5),但在验证样本中,加权回归1却明显优于加权回归2。而同时满足建模样本拟合结果最优和验证样本检验结果最优的组合中,只有加权回归1。两个树种地上部分各分量生物量,模型拟合结果和适用性,均为干材最优,树叶最差、树枝和树皮居中,样本分类、模型类型和加权最小二乘方法对干材生物量的影响,规律和地上部分总生物量相同;样本分类、模型类型和加权最小二乘方法的最优组合,用验证样本检验的结果,总相对误差树枝不超过±10.0%,树皮不超过±5.0%,树叶马尾松不超过±30.0%,落叶松不超过±20.0%。两个树种地下部分(根)生物量,样本按龄组分类方法,模型拟合结果最优,与材积相容的模型总体上优于与地上部分总生物量相容模型。     

The performance of risk prediction models

For medical decision making and patient information, predictions of future status variables play an important role. Risk prediction models can be derived with many different statistical approaches. To compare them, measures of predictive performance are derived from ROC methodology and from probability forecasting theory. These tools can be applied to assess single markers, multivariable regression models and complex model selection algorithms. This article provides a systematic review of the modern way of assessing risk prediction models. Particular attention is put on proper benchmarks and resampling techniques that are important for the interpretation of measured performance. All methods are illustrated with data from a clinical study in head and neck cancer patients.     

Not seeing the forest for the trees: Generalised linear model out-performs random forest in species distribution modelling for Southeast Asian felids

Species Distribution Models (SDMs) are a powerful tool to derive habitat suitability predictions relating species occurrence data with habitat features. Two of the most frequently applied algorithms to model species-habitat relationships are Generalised Linear Models (GLM) and Random Forest (RF). The former is a parametric regression model providing functional models with direct interpretability. The latter is a machine learning non-parametric algorithm, more tolerant than other approaches in its assumptions, which has often been shown to outperform parametric algorithms. Other approaches have been developed to produce robust SDMs, like training data bootstrapping and spatial scale optimisation. Using felid presence-absence data from three study regions in Southeast Asia (mainland, Borneo and Sumatra), we tested the performances of SDMs by implementing four modelling frameworks: GLM and RF with bootstrapped and non-bootstrapped training data. With Mantel and ANOVA tests we explored how the four combinations of algorithms and bootstrapping influenced SDMs and their predictive performances. Additionally, we tested how scale-optimisation responded to species' size, taxonomic associations (species and genus), study area and algorithm. We found that choice of algorithm had strong effect in determining the differences between SDMs' spatial predictions, while bootstrapping had no effect. Additionally, algorithm followed by study area and species, were the main factors driving differences in the spatial scales identified. SDMs trained with GLM showed higher predictive performance, however, ANOVA tests revealed that algorithm had significant effect only in explaining the variance observed in sensitivity and specificity and, when interacting with bootstrapping, in Percent Correctly Classified (PCC). Bootstrapping significantly explained the variance in specificity, PCC and True Skills Statistics (TSS). Our results suggest that there are systematic differences in the scales identified and in the predictions produced by GLM vs. RF, but that neither approach was consistently better than the other. The divergent predictions and inconsistent predictive abilities suggest that analysts should not assume machine learning is inherently superior and should test multiple methods. Our results have strong implications for SDM development, revealing the inconsistencies introduced by the choice of algorithm on scale optimisation, with GLM selecting broader scales than RF.     

Performance of several variable-selection methods applied to real ecological data

I evaluated the predictive ability of statistical models obtained by applying seven methods of variable selection to 12 ecological and environmental data sets. Cross-validation, involving repeated splits of each data set into training and validation subsets, was used to obtain honest estimates of predictive ability that could be fairly compared among methods. There was surprisingly little difference in predictive ability among five methods based on multiple linear regression. Stepwise methods performed similarly to exhaustive algorithms for subset selection, and the choice of criterion for comparing models (Akaike's information criterion, Schwarz's Bayesian information criterion or F statistics) had little effect on predictive ability. For most of the data sets, two methods based on regression trees yielded models with substantially lower predictive ability. I argue that there is no 'best' method of variable selection and that any of the regression-based approaches discussed here is capable of yielding useful predictive models.     

Sensitivity of predictive species distribution models to change in grain size

Predictive species distribution modelling (SDM) has become an essential tool in biodiversity conservation and management. The choice of grain size (resolution) of environmental layers used in modelling is one important factor that may affect predictions. We applied 10 distinct modelling techniques to presence-only data for 50 species in five different regions, to test whether: (1) a 10-fold coarsening of resolution affects predictive performance of SDMs, and (2) any observed effects are dependent on the type of region, modelling technique, or species considered. Results show that a 10 times change in grain size does not severely affect predictions from species distribution models. The overall trend is towards degradation of model performance, but improvement can also be observed. Changing grain size does not equally affect models across regions, techniques, and species types. The strongest effect is on regions and species types, with tree species in the data sets (regions) with highest locational accuracy being most affected. Changing grain size had little influence on the ranking of techniques: boosted regression trees remain best at both resolutions. The number of occurrences used for model training had an important effect, with larger sample sizes resulting in better models, which tended to be more sensitive to grain. Effect of grain change was only noticeable for models reaching sufficient performance and/or with initial data that have an intrinsic error smaller than the coarser grain size.     

No Longer Confidential: Estimating the Confidence of Individual Regression Predictions

Quantitative predictions in computational life sciences are often based on regression models. The advent of machine learning has led to highly accurate regression models that have gained widespread acceptance. While there are statistical methods available to estimate the global performance of regression models on a test or training dataset, it is often not clear how well this performance transfers to other datasets or how reliable an individual prediction is–a fact that often reduces a user’s trust into a computational method. In analogy to the concept of an experimental error, we sketch how estimators for individual prediction errors can be used to provide confidence intervals for individual predictions. Two novel statistical methods, named CONFINE and CONFIVE, can estimate the reliability of an individual prediction based on the local properties of nearby training data. The methods can be applied equally to linear and non-linear regression methods with very little computational overhead. We compare our confidence estimators with other existing confidence and applicability domain estimators on two biologically relevant problems (MHC–peptide binding prediction and quantitative structure-activity relationship (QSAR)). Our results suggest that the proposed confidence estimators perform comparable to or better than previously proposed estimation methods. Given a sufficient amount of training data, the estimators exhibit error estimates of high quality. In addition, we observed that the quality of estimated confidence intervals is predictable. We discuss how confidence estimation is influenced by noise, the number of features, and the dataset size. Estimating the confidence in individual prediction in terms of error intervals represents an important step from plain, non-informative predictions towards transparent and interpretable predictions that will help to improve the acceptance of computational methods in the biological community.     

Modeling Dynamic Systems with Efficient Ensembles of Process-Based Models

Ensembles are a well established machine learning paradigm, leading to accurate and robust models, predominantly applied to predictive modeling tasks. Ensemble models comprise a finite set of diverse predictive models whose combined output is expected to yield an improved predictive performance as compared to an individual model. In this paper, we propose a new method for learning ensembles of process-based models of dynamic systems. The process-based modeling paradigm employs domain-specific knowledge to automatically learn models of dynamic systems from time-series observational data. Previous work has shown that ensembles based on sampling observational data (i.e., bagging and boosting), significantly improve predictive performance of process-based models. However, this improvement comes at the cost of a substantial increase of the computational time needed for learning. To address this problem, the paper proposes a method that aims at efficiently learning ensembles of process-based models, while maintaining their accurate long-term predictive performance. This is achieved by constructing ensembles with sampling domain-specific knowledge instead of sampling data. We apply the proposed method to and evaluate its performance on a set of problems of automated predictive modeling in three lake ecosystems using a library of process-based knowledge for modeling population dynamics. The experimental results identify the optimal design decisions regarding the learning algorithm. The results also show that the proposed ensembles yield significantly more accurate predictions of population dynamics as compared to individual process-based models. Finally, while their predictive performance is comparable to the one of ensembles obtained with the state-of-the-art methods of bagging and boosting, they are substantially more efficient.     

Plant Species Composition Can Be Used as a Proxy to Predict Methane Emissions in Peatland Ecosystems After Land-Use Changes

Land-use change in peatlands affects important drivers of CH₄ emission such as groundwater level and nutrient availability. Due to the high spatial and temporal variability of such environmental drivers, it is hard to make good predictions of CH₄ emissions in the context of land-use changes. Here, we used plant species composition as a stable integrator of environmental drivers of CH₄ emissions. We used weighted averaging regression and calibration to make a direct link between plant species composition and CH₄ flux in an effort to predict values of CH₄ emission for a land-use gradient in two extensive peatland sites. Our predicted CH₄ emissions showed good fit with observed values. Additionally, we showed that a quick characterization of vegetation composition, by the dominant species only, provides equally good predictions of CH₄ emissions. The use of mean groundwater level alone for predicting emissions showed the same predictive power as our models. However, water level showed strong variability in time. Furthermore, the inverse relationship between water level and CH₄ emission can lead to higher errors in predictions at sites with higher CH₄ emission. The performance of our model was comparable with those of mechanistic models developed for natural wetland ecosystems. However, such mechanistic models require complex input parameters that are rarely available. We propose the use of plant species composition as a simple and effective alternative for deriving predictions of CH₄ emissions in peatlands in the context of land-use change.     

Selecting from correlated climate variables: a major source of uncertainty for predicting species distributions under climate change

Correlative species distribution models are frequently used to predict species’ range shifts under climate change. However, climate variables often show high collinearity and most statistical approaches require the selection of one among strongly correlated variables. When causal relationships between species presence and climate parameters are unknown, variable selection is often arbitrary, or based on predictive performance under current conditions. While this should only marginally affect current range predictions, future distributions may vary considerably when climate parameters do not change in concert. We investigated this source of uncertainty using four highly correlated climate variables together with a constant set of landscape variables in order to predict current (2010) and future (2050) distributions of four mountain bird species in central Europe. Simulating different parameterization decisions, we generated a) four models including each of the climate variables singly, b) a model taking advantage of all variables simultaneously and c) an un‐weighted average of the predictions of a). We compared model accuracy under current conditions, predicted distributions under four scenarios of climate change, and – for one species – evaluated back‐projections using historical occurrence data. Although current and future variable‐correlations remained constant, and the models’ accuracy under contemporary conditions did not differ, future range predictions varied considerably in all climate change scenarios. Averaged models and models containing all climate variables simultaneously produced intermediate predictions; the latter, however, performed best in back‐projections. This pattern, consistent across different modelling methods, indicates a benefit from including multiple climate predictors in ambiguous situations. Variable selection proved to be an important source of uncertainty for future range predictions, difficult to control using contemporary information. Small, but diverging changes of climate variables, masked by constant overall correlation patterns, can cause substantial differences between future range predictions which need to be accounted for, particularly when outcomes are intended for conservation decisions.     

GLM versus CCA spatial modeling of plant species distribution

Despite the variety of statistical methods available for static modeling of plant distribution, few studies directly compare methods on a common data set. In this paper, the predictive power of Generalized Linear Models (GLM) versus Canonical Correspondence Analysis (CCA) models of plant distribution in the Spring Mountains of Nevada, USA, are compared. Results show that GLM models give better predictions than CCA models because a species-specific subset of explanatory variables can be selected in GLM, while in CCA, all species are modeled using the same set of composite environmental variables (axes). Although both techniques can be readily ported to a Geographical Information System (GIS), CCA models are more readily implemented for many species at once. Predictions from both techniques rank the species models in the same order of quality; i.e. a species whose distribution is well modeled by GLM is also well modeled by CCA and vice-versa. In both cases, species for which model predictions have the poorest accuracy are either disturbance or fire related, or species for which too few observations were available to calibrate and evaluate the model. Each technique has its advantages and drawbacks. In general GLM will provide better species specific-models, but CCA will provide a broader overview of multiple species, diversity, and plant communities.     

Non-stationarity and local approaches to modelling the distributions of wildlife

Despite a growing interest in species distribution modelling, relatively little attention has been paid to spatial autocorrelation and non-stationarity. Both spatial autocorrelation (the tendency for adjacent locations to be more similar than distant ones) and non-stationarity (the variation in modelled relationships over space) are likely to be common properties of ecological systems. This paper focuses on non-stationarity and uses two local techniques, geographically weighted regression (GWR) and varying coefficient modelling (VCM), to assess its impact on model predictions. We extend two published studies, one on the presence–absence of calandra larks in Spain and the other on bird species richness in Britain, to compare GWR and VCM with the more usual global generalized linear modelling (GLM) and generalized additive modelling (GAM). For the calandra lark data, GWR and VCM produced better-fitting models than GLM or GAM. VCM in particular gave significantly reduced spatial autocorrelation in the model residuals. GWR showed that individual predictors became stationary at different spatial scales, indicating that distributions are influenced by ecological processes operating over multiple scales. VCM was able to predict occurrence accurately on independent data from the same geographical area as the training data but not beyond, whereas the GAM produced good results on all areas. Individual predictions from the local methods often differed substantially from the global models. For the species richness data, VCM and GWR produced far better predictions than ordinary regression. Our analyses suggest that modellers interpolating data to produce maps for practical actions (e.g. conservation) should consider local methods, whereas they should not be used for extrapolation to new areas. We argue that local methods are complementary to global methods, revealing details of habitat associations and data properties which global methods average out and miss.