Validation of discrete time-to-event prediction models in the presence of competing risks

Clinical prediction models play a key role in risk stratification, therapy assignment and many other fields of medical decision making. Before they can enter clinical practice, their usefulness has to be demonstrated using systematic validation. Methods to assess their predictive performance have been proposed for continuous, binary, and time-to-event outcomes, but the literature on validation methods for discrete time-to-event models with competing risks is sparse. The present paper tries to fill this gap and proposes new methodology to quantify discrimination, calibration, and prediction error (PE) for discrete time-to-event outcomes in the presence of competing risks. In our case study, the goal was to predict the risk of ventilator-associated pneumonia (VAP) attributed to Pseudomonas aeruginosa in intensive care units (ICUs). Competing events are extubation, death, and VAP due to other bacteria. The aim of this application is to validate complex prediction models developed in previous work on more recently available validation data.     

A comparison of the beta-geometric model with landmarking for dynamic prediction of time to pregnancy

We conducted a simulation study to compare two methods that have been recently used in clinical literature for the dynamic prediction of time to pregnancy. The first is landmarking, a semi-parametric method where predictions are updated as time progresses using the patient subset still at risk at that time point. The second is the beta-geometric model that updates predictions over time from a parametric model estimated on all data and is specific to applications with a discrete time to event outcome. The beta-geometric model introduces unobserved heterogeneity by modelling the chance of an event per discrete time unit according to a beta distribution. Due to selection of patients with lower chances as time progresses, the predicted probability of an event decreases over time. Both methods were recently used to develop models predicting the chance to conceive naturally. The advantages, disadvantages and accuracy of these two methods are unknown. We simulated time-to-pregnancy data according to different scenarios. We then compared the two methods by the following out-of-sample metrics: bias and root mean squared error in the average prediction, root mean squared error in individual predictions, Brier score and c statistic. We consider different scenarios including data-generating mechanisms for which the models are misspecified. We applied the two methods on a clinical dataset comprising 4999 couples. Finally, we discuss the pros and cons of the two methods based on our results and present recommendations for use of either of the methods in different settings and (effective) sample sizes.     

Combining point‐process and landscape vegetation models to predict large herbivore distributions in space and time—A case study of Rupicapra rupicapra

Aim

When modelling the distribution of animals under current and future conditions, both their response to environmental constraints and their resources’ response to these environmental constraints need to be taken into account. Here, we develop a framework to predict the distribution of large herbivores under global change, while accounting for changes in their main resources. We applied it to Rupicapra rupicapra, the chamois of the European Alps.

Location

The Bauges Regional Park (French Alps).

Methods

We built sixteen plant functional groups (PFGs) that account for the chamois’ diet (estimated from sequenced environmental DNA found in the faeces), climatic requirements, dispersal limitations, successional stage and interaction for light. These PFGs were then simulated using a dynamic vegetation model, under current and future climatic conditions up to 2100. Finally, we modelled the spatial distribution of the chamois under both current and future conditions using a point‐process model applied to either climate‐only variables or climate and simulated vegetation structure variables.

Results

Both the climate‐only and the climate and vegetation models successfully predicted the current distribution of the chamois species. However, when applied into the future, the predictions differed widely. While the climate‐only models predicted an 80% decrease in total species occupancy, including vegetation structure and plant resources for chamois in the model provided more optimistic predictions because they account for the transient dynamics of the vegetation (?20% in species occupancy).

Main conclusions

Applying our framework to the chamois shows that the inclusion of ecological mechanisms (i.e., plant resources) produces more realistic predictions under current conditions and should prove useful for anticipating future impacts. We have shown that discounting the pure effects of vegetation on chamois might lead to overpessimistic predictions under climate change. Our approach paves the way for improved synergies between different fields to produce biodiversity scenarios.

     

An evaluation of behavior inferences from Bayesian state‐space models: A case study with the Pacific walrus

State‐space models offer researchers an objective approach to modeling complex animal location data sets, and state‐space model behavior classifications are often assumed to have a link to animal behavior. In this study, we evaluated the behavioral classification accuracy of a Bayesian state‐space model in Pacific walruses using Argos satellite tags with sensors to detect animal behavior in real time. We fit a two‐state discrete‐time continuous‐space Bayesian state‐space model to data from 306 Pacific walruses tagged in the Chukchi Sea. We matched predicted locations and behaviors from the state‐space model (resident, transient behavior) to true animal behavior (foraging, swimming, hauled out) and evaluated classification accuracy with kappa statistics (κ) and root mean square error (RMSE). In addition, we compared biased random bridge utilization distributions generated with resident behavior locations to true foraging behavior locations to evaluate differences in space use patterns. Results indicated that the two‐state model fairly classified true animal behavior (0.06 ≤ κ ≤ 0.26, 0.49 ≤ RMSE ≤ 0.59). Kernel overlap metrics indicated utilization distributions generated with resident behavior locations were generally smaller than utilization distributions generated with true foraging behavior locations. Consequently, we encourage researchers to carefully examine parameters and priors associated with behaviors in state‐space models, and reconcile these parameters with the study species and its expected behaviors.