Cannot see the random forest for the decision trees: selecting predictive models for restoration ecology

Improving predictions of restoration outcomes is increasingly important to resource managers for accountability and adaptive management, yet there is limited guidance for selecting a predictive model from the multitude available. The goal of this article was to identify an optimal predictive framework for restoration ecology using 11 modeling frameworks (including machine learning, inferential, and ensemble approaches) and three data groups (field data, geographic data [GIS], and a combination thereof). We test this approach with a dataset from a large postfire sagebrush reestablishment project in the Great Basin, U.S.A. Predictive power varied among models and data groups, ranging from 58% to 79% accuracy. Finer‐scale field data generally had the greatest predictive power, although GIS data were present in the best models overall. An ensemble prediction computed from the 10 models parameterized to field data was well above average for accuracy but was outperformed by others that prioritized model parsimony by selecting predictor variables based on rankings of their importance among all candidate models. The variation in predictive power among a suite of modeling frameworks underscores the importance of a model comparison and refinement approach that evaluates multiple models and data groups, and selects variables based on their contribution to predictive power. The enhanced understanding of factors influencing restoration outcomes accomplished by this framework has the potential to aid the adaptive management process for improving future restoration outcomes.     

Defining and detecting structural sensitivity in biological models: developing a new framework

When we construct mathematical models to represent biological systems, there is always uncertainty with regards to the model specification—whether with respect to the parameters or to the formulation of model functions. Sometimes choosing two different functions with close shapes in a model can result in substantially different model predictions: a phenomenon known in the literature as structural sensitivity, which is a significant obstacle to improving the predictive power of biological models. In this paper, we revisit the general definition of structural sensitivity, compare several more specific definitions and discuss their usefulness for the construction and analysis of biological models. Then we propose a general approach to reveal structural sensitivity with regards to certain system properties, which considers infinite-dimensional neighbourhoods of the model functions: a far more powerful technique than the conventional approach of varying parameters for a fixed functional form. In particular, we suggest a rigorous method to unearth sensitivity with respect to the local stability of systems’ equilibrium points. We present a method for specifying the neighbourhood of a general unknown function with \(n\) inflection points in terms of a finite number of local function properties, and provide a rigorous proof of its completeness. Using this powerful result, we implement our method to explore sensitivity in several well-known multicomponent ecological models and demonstrate the existence of structural sensitivity in these models. Finally, we argue that structural sensitivity is an important intrinsic property of biological models, and a direct consequence of the complexity of the underlying real systems.     

Indirect determination of trabecular bone effective tissue failure properties using micro-finite element simulations

Trabecular bone strength is marked not only by the onset of local yielding, but also by post-yield behavior. To study and predict trabecular bone elastic and yield properties, micro-finite element (micro-FE) models were successfully applied. However, trabecular bone strength predictions require micro-FE models incorporating post-yield behavior of trabecular bone tissue. Due to experimental difficulties, such data is currently not available. Here we used micro-FE modeling to determine failure behavior of trabecular bone tissue indirectly, by iteratively fitting FE simulation to experimental results. Failure parameters were fitted to an isotropic plasticity model based on Hill's yield function, using materially and geometrically nonlinear micro-FE models of seven bovine trabecular bone specimens. The predictive value of the averaged effective tissue properties was subsequently tested. The results showed that compression softening had to be included on the tissue level in order to accurately describe the apparent-level behavior of the bone specimens. A sensitivity study revealed that the simulated response was less sensitive to variations in the post-yield properties of the bone tissue than variations in the elastic and yield properties. Due to fitting of the tissue properties, apparent-level behavior could be accurately reproduced for each specimen separately. Predictions based on the averaged and fixed tissue properties were less accurate, due to inter-specimen variations in the tissue properties.     

Integrating across neuroimaging modalities boosts prediction accuracy of cognitive ability

Variation in cognitive ability arises from subtle differences in underlying neural architecture. Understanding and predicting individual variability in cognition from the differences in brain networks requires harnessing the unique variance captured by different neuroimaging modalities. Here we adopted a multi-level machine learning approach that combines diffusion, functional, and structural MRI data from the Human Connectome Project (N = 1050) to provide unitary prediction models of various cognitive abilities: global cognitive function, fluid intelligence, crystallized intelligence, impulsivity, spatial orientation, verbal episodic memory and sustained attention. Out-of-sample predictions of each cognitive score were first generated using a sparsity-constrained principal component regression on individual neuroimaging modalities. These individual predictions were then aggregated and submitted to a LASSO estimator that removed redundant variability across channels. This stacked prediction led to a significant improvement in accuracy, relative to the best single modality predictions (approximately 1% to more than 3% boost in variance explained), across a majority of the cognitive abilities tested. Further analysis found that diffusion and brain surface properties contribute the most to the predictive power. Our findings establish a lower bound to predict individual differences in cognition using multiple neuroimaging measures of brain architecture, both structural and functional, quantify the relative predictive power of the different imaging modalities, and reveal how each modality provides unique and complementary information about individual differences in cognitive function.     

Lumbar spine finite element model for healthy subjects: development and validation

Finite element (FE) method is a proven powerful and efficient tool to study the biomechanics of the human lumbar spine. However, due to the large inter-subject variability of geometries and material properties in human lumbar spines, concerns existed on the accuracy and predictive power of one single deterministic FE model with one set of spinal geometry and material properties. It was confirmed that the combined predictions (median or mean value) of several distinct FE models can be used as an improved prediction of behavior of human lumbar spine under identical loading and boundary conditions. In light of this improved prediction, five FE models (L1-L5 spinal levels) of the human lumbar spine were developed based on five healthy living subjects with identical modeling method. The five models were extensively validated through experimental and computational results in the literature. Mesh convergence and material sensitivity analysis were also conducted. We have shown that the results from the five FE models developed in this paper were consistent with the experimental data and simulation results from the existing literature. The validated modeling method introduced in this study can be used in modeling dysfunctional lumber spines such as disc degeneration and scoliosis in future work.     

Sources of Variability in a Synthetic Gene Oscillator

Synthetic gene oscillators are small, engineered genetic circuits that produce periodic variations in target protein expression. Like other gene circuits, synthetic gene oscillators are noisy and exhibit fluctuations in amplitude and period. Understanding the origins of such variability is key to building predictive models that can guide the rational design of synthetic circuits. Here, we developed a method for determining the impact of different sources of noise in genetic oscillators by measuring the variability in oscillation amplitude and correlations between sister cells. We first used a combination of microfluidic devices and time-lapse fluorescence microscopy to track oscillations in cell lineages across many generations. We found that oscillation amplitude exhibited high cell-to-cell variability, while sister cells remained strongly correlated for many minutes after cell division. To understand how such variability arises, we constructed a computational model that identified the impact of various noise sources across the lineage of an initial cell. When each source of noise was appropriately tuned the model reproduced the experimentally observed amplitude variability and correlations, and accurately predicted outcomes under novel experimental conditions. Our combination of computational modeling and time-lapse data analysis provides a general way to examine the sources of variability in dynamic gene circuits.     

Information Sets and Excess Zeros in Random Effects Modeling of Longitudinal Data

Marginal regression via generalized estimating equations is widely used in biostatistics to model longitudinal data from subjects whose outcomes and covariates are observed at several time points. In this paper we consider two issues that have been raised in the literature concerning the marginal regression approach. The first is that even though the past history may be predictive of outcome, the marginal approach does not use this history. Although marginal regression has the flexibility of allowing between-subject variations in the observation times, it may lose substantial prediction power in comparison with the transitional modeling approach that relates the responses to the covariate and outcome histories. We address this issue by using the concept of “information sets” for prediction to generalize the “partly conditional mean” approach of Pepe and Couper (J. Am. Stat. Assoc. 92:991–998, 1997). This modeling approach strikes a balance between the flexibility of the marginal approach and the predictive power of transitional modeling. Another issue is the problem of excess zeros in the outcomes over what the underlying model for marginal regression implies. We show how our predictive modeling approach based on information sets can be readily modified to handle the excess zeros in the longitudinal time series. By synthesizing the marginal, transitional, and mixed effects modeling approaches in a predictive framework, we also discuss how their respective advantages can be retained while their limitations can be circumvented for modeling longitudinal data.     

Fine scale prediction of ecological community composition using a two-step sequential Machine Learning ensemble

Prediction is one of the last frontiers in ecology. Indeed, predicting fine-scale species composition in natural systems is a complex challenge as multiple abiotic and biotic processes operate simultaneously to determine local species abundances. On the one hand, species intrinsic performance and their tolerance limits to different abiotic pressures modulate species abundances. On the other hand, there is growing recognition that species interactions play an equally important role in limiting or promoting such abundances within ecological communities. Here, we present a joint effort between ecologists and data scientists to use data-driven models to predict species abundances using reasonably easy to obtain data. We propose a sequential data-driven modeling approach that in a first step predicts the potential species abundances based on abiotic variables, and in a second step uses these predictions to model the realized abundances once accounting for species competition. Using a curated data set over five years we predict fine-scale species abundances in a highly diverse annual plant community. Our models show a remarkable spatial predictive accuracy using only easy-to-measure variables in the field, yet such predictive power is lost when temporal dynamics are taken into account. This result suggests that predicting future abundances requires longer time series analysis to capture enough variability. In addition, we show that these data-driven models can also suggest how to improve mechanistic models by adding missing variables that affect species performance such as particular soil conditions (e.g. carbonate availability in our case). Robust models for predicting fine-scale species composition informed by the mechanistic understanding of the underlying abiotic and biotic processes can be a pivotal tool for conservation, especially given the human-induced rapid environmental changes we are experiencing. This objective can be achieved by promoting the knowledge gained with classic modelling approaches in ecology and recently developed data-driven models.     

The Logic of EGFR/ErbB Signaling: Theoretical Properties and Analysis of High-Throughput Data

The epidermal growth factor receptor (EGFR) signaling pathway is probably the best-studied receptor system in mammalian cells, and it also has become a popular example for employing mathematical modeling to cellular signaling networks. Dynamic models have the highest explanatory and predictive potential; however, the lack of kinetic information restricts current models of EGFR signaling to smaller sub-networks. This work aims to provide a large-scale qualitative model that comprises the main and also the side routes of EGFR/ErbB signaling and that still enables one to derive important functional properties and predictions. Using a recently introduced logical modeling framework, we first examined general topological properties and the qualitative stimulus-response behavior of the network. With species equivalence classes, we introduce a new technique for logical networks that reveals sets of nodes strongly coupled in their behavior. We also analyzed a model variant which explicitly accounts for uncertainties regarding the logical combination of signals in the model. The predictive power of this model is still high, indicating highly redundant sub-structures in the network. Finally, one key advance of this work is the introduction of new techniques for assessing high-throughput data with logical models (and their underlying interaction graph). By employing these techniques for phospho-proteomic data from primary hepatocytes and the HepG2 cell line, we demonstrate that our approach enables one to uncover inconsistencies between experimental results and our current qualitative knowledge and to generate new hypotheses and conclusions. Our results strongly suggest that the Rac/Cdc42 induced p38 and JNK cascades are independent of PI3K in both primary hepatocytes and HepG2. Furthermore, we detected that the activation of JNK in response to neuregulin follows a PI3K-dependent signaling pathway.     

Sensitivity analysis of the tree distribution model Phenofit to climatic input characteristics: implications for climate impact assessment

Species distributions are already affected by climate change. Forecasting their long‐term evolution requires models with thoroughly assessed validation. Our aim here is to demonstrate that the sensitivity of such models to climate input characteristics may complicate their validation and introduce uncertainties in their predictions. In this study, we conducted a sensitivity analysis of a process‐based tree distribution model Phenofit to climate input characteristics. This analysis was conducted for two North American trees which differ greatly in their distribution and eight different types of climate input for the historic period which differ in their spatial (local or gridded data) and temporal (daily vs. monthly) resolution as well as their type (locally recorded, extrapolated or simulated by General Circulation Models). We show that the climate data resolution (spatial and temporal) and their type, highly affect the model predictions. The sensitivity analysis also revealed, the importance, for global climate change impact assessment, of (i) the daily variability of temperatures in modeling the biological processes shaping species distribution, (ii) climate data at high latitudes and elevations and (iii) climate data with high spatial resolution.     

Predicted residue–residue contacts can help the scoring of 3D models

During the 7th Critical Assessment of Protein Structure Prediction (CASP7) experiment, it was suggested that the real value of predicted residue–residue contacts might lie in the scoring of 3D model structures. Here, we have carried out a detailed reassessment of the contact predictions made during the recent CASP8 experiment to determine whether predicted contacts might aid in the selection of close‐to‐native structures or be a useful tool for scoring 3D structural models. We used the contacts predicted by the CASP8 residue–residue contact prediction groups to select models for each target domain submitted to the experiment. We found that the information contained in the predicted residue–residue contacts would probably have helped in the selection of 3D models in the free modeling regime and over the harder comparative modeling targets. Indeed, in many cases, the models selected using just the predicted contacts had better GDT‐TS scores than all but the best 3D prediction groups. Despite the well‐known low accuracy of residue–residue contact predictions, it is clear that the predictive power of contacts can be useful in 3D model prediction strategies. Proteins 2010. © 2010 Wiley‐Liss, Inc.     

Testing projected wild bee distributions in agricultural habitats: predictive power depends on species traits and habitat type

Species distribution models (SDM) are increasingly used to understand the factors that regulate variation in biodiversity patterns and to help plan conservation strategies. However, these models are rarely validated with independently collected data and it is unclear whether SDM performance is maintained across distinct habitats and for species with different functional traits. Highly mobile species, such as bees, can be particularly challenging to model. Here, we use independent sets of occurrence data collected systematically in several agricultural habitats to test how the predictive performance of SDMs for wild bee species depends on species traits, habitat type, and sampling technique. We used a species distribution modeling approach parametrized for the Netherlands, with presence records from 1990 to 2010 for 193 Dutch wild bees. For each species, we built a Maxent model based on 13 climate and landscape variables. We tested the predictive performance of the SDMs with independent datasets collected from orchards and arable fields across the Netherlands from 2010 to 2013, using transect surveys or pan traps. Model predictive performance depended on species traits and habitat type. Occurrence of bee species specialized in habitat and diet was better predicted than generalist bees. Predictions of habitat suitability were also more precise for habitats that are temporally more stable (orchards) than for habitats that suffer regular alterations (arable), particularly for small, solitary bees. As a conservation tool, SDMs are best suited to modeling rarer, specialist species than more generalist and will work best in long‐term stable habitats. The variability of complex, short‐term habitats is difficult to capture in such models and historical land use generally has low thematic resolution. To improve SDMs’ usefulness, models require explanatory variables and collection data that include detailed landscape characteristics, for example, variability of crops and flower availability. Additionally, testing SDMs with field surveys should involve multiple collection techniques.     

“Essentially,all models are wrong,but some are useful”<Emphasis Type="Italic">—</Emphasis>a cross-disciplinary agenda for building useful models in cell biology and biophysics

Intuition alone often fails to decipher the mechanisms underlying the experimental data in Cell Biology and Biophysics, and mathematical modeling has become a critical tool in these fields. However, mathematical modeling is not as widespread as it could be, because experimentalists and modelers often have difficulties communicating with each other, and are not always on the same page about what a model can or should achieve. Here, we present a framework to develop models that increase the understanding of the mechanisms underlying one’s favorite biological system. Development of the most insightful models starts with identifying a good biological question in light of what is known and unknown in the field, and determining the proper level of details that are sufficient to address this question. The model should aim not only to explain already available data, but also to make predictions that can be experimentally tested. We hope that both experimentalists and modelers who are driven by mechanistic questions will find these guidelines useful to develop models with maximum impact in their field.     

Predicting Environmental Suitability for a Rare and Threatened Species (Lao Newt,Laotriton laoensis) Using Validated Species Distribution Models

The Lao newt (Laotriton laoensis) is a recently described species currently known only from northern Laos. Little is known about the species, but it is threatened as a result of overharvesting. We integrated field survey results with climate and altitude data to predict the geographic distribution of this species using the niche modeling program Maxent, and we validated these predictions by using interviews with local residents to confirm model predictions of presence and absence. The results of the validated Maxent models were then used to characterize the environmental conditions of areas predicted suitable for L. laoensis. Finally, we overlaid the resulting model with a map of current national protected areas in Laos to determine whether or not any land predicted to be suitable for this species is coincident with a national protected area. We found that both area under the curve (AUC) values and interview data provided strong support for the predictive power of these models, and we suggest that interview data could be used more widely in species distribution niche modeling. Our results further indicated that this species is mostly likely geographically restricted to high altitude regions (i.e., over 1,000 m elevation) in northern Laos and that only a minute fraction of suitable habitat is currently protected. This work thus emphasizes that increased protection efforts, including listing this species as endangered and the establishment of protected areas in the region predicted to be suitable for L. laoensis, are urgently needed.     

Validation of species–climate impact models under climate change

Increasing concern over the implications of climate change for biodiversity has led to the use of species–climate envelope models to project species extinction risk under climate‐change scenarios. However, recent studies have demonstrated significant variability in model predictions and there remains a pressing need to validate models and to reduce uncertainties. Model validation is problematic as predictions are made for events that have not yet occurred. Resubstituition and data partitioning of present‐day data sets are, therefore, commonly used to test the predictive performance of models. However, these approaches suffer from the problems of spatial and temporal autocorrelation in the calibration and validation sets. Using observed distribution shifts among 116 British breeding‐bird species over the past ～20 years, we are able to provide a first independent validation of four envelope modelling techniques under climate change. Results showed good to fair predictive performance on independent validation, although rules used to assess model performance are difficult to interpret in a decision‐planning context. We also showed that measures of performance on nonindependent data provided optimistic estimates of models' predictive ability on independent data. Artificial neural networks and generalized additive models provided generally more accurate predictions of species range shifts than generalized linear models or classification tree analysis. Data for independent model validation and replication of this study are rare and we argue that perfect validation may not in fact be conceptually possible. We also note that usefulness of models is contingent on both the questions being asked and the techniques used. Implementations of species–climate envelope models for testing hypotheses and predicting future events may prove wrong, while being potentially useful if put into appropriate context.     

A joint fairness model with applications to risk predictions for underrepresented populations

In data collection for predictive modeling, underrepresentation of certain groups, based on gender, race/ethnicity, or age, may yield less accurate predictions for these groups. Recently, this issue of fairness in predictions has attracted significant attention, as data-driven models are increasingly utilized to perform crucial decision-making tasks. Existing methods to achieve fairness in the machine learning literature typically build a single prediction model in a manner that encourages fair prediction performance for all groups. These approaches have two major limitations: (i) fairness is often achieved by compromising accuracy for some groups; (ii) the underlying relationship between dependent and independent variables may not be the same across groups. We propose a joint fairness model (JFM) approach for logistic regression models for binary outcomes that estimates group-specific classifiers using a joint modeling objective function that incorporates fairness criteria for prediction. We introduce an accelerated smoothing proximal gradient algorithm to solve the convex objective function, and present the key asymptotic properties of the JFM estimates. Through simulations, we demonstrate the efficacy of the JFM in achieving good prediction performance and across-group parity, in comparison with the single fairness model, group-separate model, and group-ignorant model, especially when the minority group's sample size is small. Finally, we demonstrate the utility of the JFM method in a real-world example to obtain fair risk predictions for underrepresented older patients diagnosed with coronavirus disease 2019 (COVID-19).     

Wrong,but useful: regional species distribution models may not be improved by range‐wide data under biased sampling

Species distribution modeling (SDM) is an essential method in ecology and conservation. SDMs are often calibrated within one country's borders, typically along a limited environmental gradient with biased and incomplete data, making the quality of these models questionable. In this study, we evaluated how adequate are national presence‐only data for calibrating regional SDMs. We trained SDMs for Egyptian bat species at two different scales: only within Egypt and at a species‐specific global extent. We used two modeling algorithms: Maxent and elastic net, both under the point‐process modeling framework. For each modeling algorithm, we measured the congruence of the predictions of global and regional models for Egypt, assuming that the lower the congruence, the lower the appropriateness of the Egyptian dataset to describe the species' niche. We inspected the effect of incorporating predictions from global models as additional predictor (“prior”) to regional models, and quantified the improvement in terms of AUC and the congruence between regional models run with and without priors. Moreover, we analyzed predictive performance improvements after correction for sampling bias at both scales. On average, predictions from global and regional models in Egypt only weakly concur. Collectively, the use of priors did not lead to much improvement: similar AUC and high congruence between regional models calibrated with and without priors. Correction for sampling bias led to higher model performance, whatever prior used, making the use of priors less pronounced. Under biased and incomplete sampling, the use of global bats data did not improve regional model performance. Without enough bias‐free regional data, we cannot objectively identify the actual improvement of regional models after incorporating information from the global niche. However, we still believe in great potential for global model predictions to guide future surveys and improve regional sampling in data‐poor regions.     

The dynamical consequences of developmental variability and demographic stochasticity for host-parasitoid interactions

Few age-structured models of species dynamics incorporate variability and uncertainty in population processes. Motivated by laboratory data for an insect and its parasitoid, we investigate whether such assumptions are appropriate when considering the population dynamics of a single species and its interaction with a natural enemy. Specifically, we examine the effects of developmental variability and demographic stochasticity on different types of cyclic dynamics predicted by traditional models. We show that predictions based on the deterministic fixed-development approach are differentially sensitive to variability and noise in key life stages. In particular, we find that the demonstration of half-generation cycles in the single-species model and the multigeneration cycles in the host-parasitoid model are sensitive to the introduction of developmental variability and noise, whereas generation cycles are robust to the intrinsic variability and uncertainty that may be found in nature.