A comparison of machine learning algorithms and covariate balance measures for propensity score matching and weighting

Propensity score matching (PSM) and propensity score weighting (PSW) are popular tools to estimate causal effects in observational studies. We address two open issues: how to estimate propensity scores and assess covariate balance. Using simulations, we compare the performance of PSM and PSW based on logistic regression and machine learning algorithms (CART; Bagging; Boosting; Random Forest; Neural Networks; naive Bayes). Additionally, we consider several measures of covariate balance (Absolute Standardized Average Mean (ASAM) with and without interactions; measures based on the quantile‐quantile plots; ratio between variances of propensity scores; area under the curve (AUC)) and assess their ability in predicting the bias of PSM and PSW estimators. We also investigate the importance of tuning of machine learning parameters in the context of propensity score methods. Two simulation designs are employed. In the first, the generating processes are inspired to birth register data used to assess the effect of labor induction on the occurrence of caesarean section. The second exploits more general generating mechanisms. Overall, among the different techniques, random forests performed the best, especially in PSW. Logistic regression and neural networks also showed an excellent performance similar to that of random forests. As for covariate balance, the simplest and commonly used metric, the ASAM, showed a strong correlation with the bias of causal effects estimators. Our findings suggest that researchers should aim at obtaining an ASAM lower than 10% for as many variables as possible. In the empirical study we found that labor induction had a small and not statistically significant impact on caesarean section.     

Developing a discrimination rule between breast cancer patients and controls using proteomics mass spectrometric data: a three-step approach

To discriminate between breast cancer patients and controls, we used a three-step approach to obtain our decision rule. First, we ranked the mass/charge values using random forests, because it generates importance indices that take possible interactions into account. We observed that the top ranked variables consisted of highly correlated contiguous mass/charge values, which were grouped in the second step into new variables. Finally, these newly created variables were used as predictors to find a suitable discrimination rule. In this last step, we compared three different methods, namely Classification and Regression Tree (CART), logistic regression and penalized logistic regression. Logistic regression and penalized logistic regression performed equally well and both had a higher classification accuracy than CART. The model obtained with penalized logistic regression was chosen as we hypothesized that this model would provide a better classification accuracy in the validation set. The solution had a good performance on the training set with a classification accuracy of 86.3%, and a sensitivity and specificity of 86.8% and 85.7%, respectively.     

The Power of the Mantel-Haenszel Test when Adjustment is by the True or Estimated Propensity Score

The power of the Mantel-Haenszel test for no treatment effect in the case of binary exposure and response variates was examined through simulation studies when subclasses were formed on the basis of the true and estimated propensity scores and by direct stratification on two continuous covariates. The power of these tests was also compared to the score test in a misspecified logistic regression model. In general adjustment by the true propensity score was most likely to reject a false null hypothesis, the score test was more likely to reject a false null hypothesis than the Mantel-Haenszel test when adjustment is by the estimated propensity score or subclassification on the covariates. There was litte difference in the observed powers of the Mantel-Haenszel tests between adjustment by the estimated propensity score and subclassification on the covariates.     

Regression trees for predicting mortality in patients with cardiovascular disease: What improvement is achieved by using ensemble‐based methods?

In biomedical research, the logistic regression model is the most commonly used method for predicting the probability of a binary outcome. While many clinical researchers have expressed an enthusiasm for regression trees, this method may have limited accuracy for predicting health outcomes. We aimed to evaluate the improvement that is achieved by using ensemble‐based methods, including bootstrap aggregation (bagging) of regression trees, random forests, and boosted regression trees. We analyzed 30‐day mortality in two large cohorts of patients hospitalized with either acute myocardial infarction (N = 16,230) or congestive heart failure (N = 15,848) in two distinct eras (1999–2001 and 2004–2005). We found that both the in‐sample and out‐of‐sample prediction of ensemble methods offered substantial improvement in predicting cardiovascular mortality compared to conventional regression trees. However, conventional logistic regression models that incorporated restricted cubic smoothing splines had even better performance. We conclude that ensemble methods from the data mining and machine learning literature increase the predictive performance of regression trees, but may not lead to clear advantages over conventional logistic regression models for predicting short‐term mortality in population‐based samples of subjects with cardiovascular disease.     

K‐Sample comparisons using propensity analysis

In this paper, we investigate K‐group comparisons on survival endpoints for observational studies. In clinical databases for observational studies, treatment for patients are chosen with probabilities varying depending on their baseline characteristics. This often results in noncomparable treatment groups because of imbalance in baseline characteristics of patients among treatment groups. In order to overcome this issue, we conduct propensity analysis and match the subjects with similar propensity scores across treatment groups or compare weighted group means (or weighted survival curves for censored outcome variables) using the inverse probability weighting (IPW). To this end, multinomial logistic regression has been a popular propensity analysis method to estimate the weights. We propose to use decision tree method as an alternative propensity analysis due to its simplicity and robustness. We also propose IPW rank statistics, called Dunnett‐type test and ANOVA‐type test, to compare 3 or more treatment groups on survival endpoints. Using simulations, we evaluate the finite sample performance of the weighted rank statistics combined with these propensity analysis methods. We demonstrate these methods with a real data example. The IPW method also allows us for unbiased estimation of population parameters of each treatment group. In this paper, we limit our discussions to survival outcomes, but all the methods can be easily modified for any type of outcomes, such as binary or continuous variables.     

Extensions of the Penalized Spline of Propensity Prediction Method of Imputation

Summary .  Little and An (2004,  Statistica Sinica   14, 949–968) proposed a penalized spline of propensity prediction (PSPP) method of imputation of missing values that yields robust model-based inference under the missing at random assumption. The propensity score for a missing variable is estimated and a regression model is fitted that includes the spline of the estimated logit propensity score as a covariate. The predicted unconditional mean of the missing variable has a double robustness (DR) property under misspecification of the imputation model. We show that a simplified version of PSPP, which does not center other regressors prior to including them in the prediction model, also has the DR property. We also propose two extensions of PSPP, namely, stratified PSPP and bivariate PSPP, that extend the DR property to inferences about conditional means. These extended PSPP methods are compared with the PSPP method and simple alternatives in a simulation study and applied to an online weight loss study conducted by Kaiser Permanente.     

Improving the Spatial Prediction of Soil Organic Carbon Stocks in a Complex Tropical Mountain Landscape by Methodological Specifications in Machine Learning Approaches

Tropical forests are significant carbon sinks and their soils’ carbon storage potential is immense. However, little is known about the soil organic carbon (SOC) stocks of tropical mountain areas whose complex soil-landscape and difficult accessibility pose a challenge to spatial analysis. The choice of methodology for spatial prediction is of high importance to improve the expected poor model results in case of low predictor-response correlations. Four aspects were considered to improve model performance in predicting SOC stocks of the organic layer of a tropical mountain forest landscape: Different spatial predictor settings, predictor selection strategies, various machine learning algorithms and model tuning. Five machine learning algorithms: random forests, artificial neural networks, multivariate adaptive regression splines, boosted regression trees and support vector machines were trained and tuned to predict SOC stocks from predictors derived from a digital elevation model and satellite image. Topographical predictors were calculated with a GIS search radius of 45 to 615 m. Finally, three predictor selection strategies were applied to the total set of 236 predictors. All machine learning algorithms—including the model tuning and predictor selection—were compared via five repetitions of a tenfold cross-validation. The boosted regression tree algorithm resulted in the overall best model. SOC stocks ranged between 0.2 to 17.7 kg m^-2, displaying a huge variability with diffuse insolation and curvatures of different scale guiding the spatial pattern. Predictor selection and model tuning improved the models’ predictive performance in all five machine learning algorithms. The rather low number of selected predictors favours forward compared to backward selection procedures. Choosing predictors due to their indiviual performance was vanquished by the two procedures which accounted for predictor interaction.     

Associations between Depressive State and Impaired Higher-Level Functional Capacity in the Elderly with Long-Term Care Requirements

Depressive state has been reported to be significantly associated with higher-level functional capacity among community-dwelling elderly. However, few studies have investigated the associations among people with long-term care requirements. We aimed to investigate the associations between depressive state and higher-level functional capacity and obtain marginal odds ratios using propensity score analyses in people with long-term care requirements. We conducted a cross-sectional study based on participants aged ≥65 years (n = 545) who were community dwelling and used outpatient care services for long-term preventive care. We measured higher-level functional capacity, depressive state, and possible confounders. Then, we estimated the marginal odds ratios (i.e., the change in odds of impaired higher-level functional capacity if all versus no participants were exposed to depressive state) by logistic models using generalized linear models with the inverse probability of treatment weighting (IPTW) for propensity score and design-based standard errors. Depressive state was used as the exposure variable and higher-level functional capacity as the outcome variable. The all absolute standardized differences after the IPTW using the propensity scores were <10% which indicated negligible differences in the mean or prevalence of the covariates between non-depressive state and depressive state. The marginal odds ratios were estimated by the logistic models with IPTW using the propensity scores. The marginal odds ratios were 2.17 (95%CI: 1.13–4.19) for men and 2.57 (95%CI: 1.26–5.26) for women. Prevention of depressive state may contribute to not only depressive state but also higher-level functional capacity.     

Cox regression with survival-time-dependent missing covariate values

Analysis with time-to-event data in clinical and epidemiological studies often encounters missing covariate values, and the missing at random assumption is commonly adopted, which assumes that missingness depends on the observed data, including the observed outcome which is the minimum of survival and censoring time. However, it is conceivable that in certain settings, missingness of covariate values is related to the survival time but not to the censoring time. This is especially so when covariate missingness is related to an unmeasured variable affected by the patient's illness and prognosis factors at baseline. If this is the case, then the covariate missingness is not at random as the survival time is censored, and it creates a challenge in data analysis. In this article, we propose an approach to deal with such survival-time-dependent covariate missingness based on the well known Cox proportional hazard model. Our method is based on inverse propensity weighting with the propensity estimated by nonparametric kernel regression. Our estimators are consistent and asymptotically normal, and their finite-sample performance is examined through simulation. An application to a real-data example is included for illustration.     

机器学习方法在基因功能注释中的应用

目前,基于计算机数学方法对基因的功能注释已成为热点及挑战,其中以机器学习方法应用最为广泛。生物信息学家不断提出有效、快速、准确的机器学习方法用于基因功能的注释,极大促进了生物医学的发展。本文就关于机器学习方法在基因功能注释的应用与进展作一综述。主要介绍几种常用的方法,包括支持向量机、k近邻算法、决策树、随机森林、神经网络、马尔科夫随机场、logistic回归、聚类算法和贝叶斯分类器,并对目前机器学习方法应用于基因功能注释时如何选择数据源、如何改进算法以及如何提高预测性能上进行讨论。     

Classification trees: An alternative non‐parametric approach for predicting species distributions

Abstract. The use of Generalized Linear Models (GLM) in vegetation analysis has been advocated to accommodate complex species response curves. This paper investigates the potential advantages of using classification and regression trees (CART), a recursive partitioning method that is free of distributional assumptions. We used multiple logistic regression (a form of GLM) and CART to predict the distribution of three major oak species in California. We compared two types of model: polynomial logistic regression models optimized to account for non‐linearity and factor interactions, and simple CART‐models. Each type of model was developed using learning data sets of 2085 and 410 sample cases, and assessed on test sets containing 2016 and 3691 cases respectively. The responses of the three species to environmental gradients were varied and often non‐homogeneous or context dependent. We tested the methods for predictive accuracy: CART‐models performed significantly better than our polynomial logistic regression models in four of the six cases considered, and as well in the two remaining cases. CART also showed a superior ability to detect factor interactions. Insight gained from CART‐models then helped develop improved parametric models. Although the probabilistic form of logistic regression results is more adapted to test theories about species responses to environmental gradients, we found that CART‐models are intuitive, easy to develop and interpret, and constitute a valuable tool for modeling species distributions.     

基于不同特征挖掘方法结合广义提升回归模型估测安徽省土壤pH

为探讨不同特征挖掘方法与广义提升回归模型相结合在数字土壤制图中的应用,本研究首先使用递归特征消除和过滤式两种特征筛选方法对环境协变量进行筛选,再分别使用原始环境协变量、筛选后的最优变量组合作为自变量,建立基于广义提升回归模型和随机森林模型的安徽省土壤pH预测模型并进行制图。结果表明: 引入两种特征挖掘方法均可有效提高广义提升回归模型和随机森林模型预测土壤pH的精度,并且可以起到降维的作用;相较于随机森林模型,广义提升回归模型的验证集预测精度略低,在训练集中,广义提升回归模型的精度却远高于随机森林模型,模型解释度高,整体效果较好;随机森林模型的主要参数ntree和mtry对于模型的影响程度较低,而不同参数对于广义提升回归模型的预测精度影响较大,不同参数组合模型精度不同,建模前需要进行调参。空间制图结果表明,安徽省土壤pH呈“南酸北碱”趋势。     

Improved procedures for estimation of disease prevalence using ranked set sampling

Ranked set sampling (RSS) is a sampling procedure that can be considerably more efficient than simple random sampling (SRS). When the variable of interest is binary, ranking of the sample observations can be implemented using the estimated probabilities of success obtained from a logistic regression model developed for the binary variable. The main objective of this study is to use substantial data sets to investigate the application of RSS to estimation of a proportion for a population that is different from the one that provides the logistic regression. Our results indicate that precision in estimation of a population proportion is improved through the use of logistic regression to carry out the RSS ranking and, hence, the sample size required to achieve a desired precision is reduced. Further, the choice and the distribution of covariates in the logistic regression model are not overly crucial for the performance of a balanced RSS procedure.     

CART peptide: central mediator of leptin-induced adipose tissue apoptosis?

Because of connections between CART peptide containing neurons and the sympathetic nervous system (SNS) and the possible role of the SNS in leptin-induced adipose apoptosis, CART may act as a downstream effector of leptin-induced adipose apoptosis. Male Sprague-Dawley rats received continuous intracerebroventricular (i.c.v.) infusion for 4 days of either artificial cerebrospinal fluid (aCSF, 12 microl/day), leptin (15 microg/day), or CART55-102 at 2.4 microg/day (CART2.4) or 9.6 microg/day (CART9.6). Food intake (FI) was decreased 10.8% for CART2.4, 41.9% for CART9.6 and 33.4% for leptin (p<0.05). CART9.6 and leptin reduced meal size and meal number. Body weight (BW) was reduced by CART9.6 (14.6%) and leptin (11.6%) (p<0.05), but not by CART2.4. CART9.6 and CART2.4, but not leptin, caused hypothermia, and CART9.6 inhibited physical activity (p<0.05). Epididymal, inguinal and retroperitoneal fat pad weights were reduced (p<0.05) by both CART treatments and leptin; CART9.6 also reduced gastrocnemius muscle weight (18.1%, p<0.05). Leptin, but not CART, increased serum free fatty acid concentrations by 31.1% (p<0.05) and increased adipose apoptosis by 48% (p<0.05). These data show that although leptin and CART55-102 have some similar actions, CART55-102 is probably not a mediator for leptin-induced adipose apoptosis in the brain.     

Sensitivity analysis for causal inference using inverse probability weighting

Evaluation of impact of potential uncontrolled confounding is an important component for causal inference based on observational studies. In this article, we introduce a general framework of sensitivity analysis that is based on inverse probability weighting. We propose a general methodology that allows both non‐parametric and parametric analyses, which are driven by two parameters that govern the magnitude of the variation of the multiplicative errors of the propensity score and their correlations with the potential outcomes. We also introduce a specific parametric model that offers a mechanistic view on how the uncontrolled confounding may bias the inference through these parameters. Our method can be readily applied to both binary and continuous outcomes and depends on the covariates only through the propensity score that can be estimated by any parametric or non‐parametric method. We illustrate our method with two medical data sets.     

Score test for missing at random or not under logistic missingness models

Missing data are frequently encountered in various disciplines and can be divided into three categories: missing completely at random (MCAR), missing at random (MAR), and missing not at random (MNAR). Valid statistical approaches to missing data depend crucially on correct identification of the underlying missingness mechanism. Although the problem of testing whether this mechanism is MCAR or MAR has been extensively studied, there has been very little research on testing MAR versus MNAR. A critical challenge that is faced when dealing with this problem is the issue of model identification under MNAR. In this paper, under a logistic model for the missing probability, we develop two score tests for the problem of whether the missingness mechanism is MAR or MNAR under a parametric model and a semiparametric location model on the regression function. The implementation of the score tests circumvents the identification issue as it requires only parameter estimation under the null MAR assumption. Our simulations and analysis of human immunodeficiency virus data show that the score tests have well-controlled type I errors and desirable powers.     

Hendra Virus Spillover is a Bimodal System Driven by Climatic Factors

Understanding environmental factors driving spatiotemporal patterns of disease can improve risk mitigation strategies. Hendra virus (HeV), discovered in Australia in 1994, spills over from bats (Pteropus sp.) to horses and thence to humans. Below latitude ? 22°, almost all spillover events to horses occur during winter, and above this latitude spillover is aseasonal. We generated a statistical model of environmental drivers of HeV spillover per month. The model reproduced the spatiotemporal pattern of spillover risk between 1994 and 2015. The model was generated with an ensemble of methods for presence–absence data (boosted regression trees, random forests and logistic regression). Presences were the locations of horse cases, and absences per spatial unit (2.7 × 2.7 km pixels without spillover) were sampled with the horse census of Queensland and New South Wales. The most influential factors indicate that spillover is associated with both cold-dry and wet conditions. Bimodal responses to several variables suggest spillover involves two systems: one above and one below a latitudinal area close to ? 22°. Northern spillovers are associated with cold-dry and wet conditions, and southern with cold-dry conditions. Biologically, these patterns could be driven by immune or behavioural changes in response to food shortage in bats and horse husbandry. Future research should look for differences in these traits between seasons in the two latitudinal regions. Based on the predicted risk patterns by latitude, we recommend enhanced preventive management for horses from March to November below latitude 22° south.     

Comparison of complex modeling strategies for prediction of a binary outcome based on a few,highly correlated predictors

Motivated by a clinical prediction problem, a simulation study was performed to compare different approaches for building risk prediction models. Robust prediction models for hospital survival in patients with acute heart failure were to be derived from three highly correlated blood parameters measured up to four times, with predictive ability having explicit priority over interpretability. Methods that relied only on the original predictors were compared with methods using an expanded predictor space including transformations and interactions. Predictors were simulated as transformations and combinations of multivariate normal variables which were fitted to the partly skewed and bimodally distributed original data in such a way that the simulated data mimicked the original covariate structure. Different penalized versions of logistic regression as well as random forests and generalized additive models were investigated using classical logistic regression as a benchmark. Their performance was assessed based on measures of predictive accuracy, model discrimination, and model calibration. Three different scenarios using different subsets of the original data with different numbers of observations and events per variable were investigated. In the investigated setting, where a risk prediction model should be based on a small set of highly correlated and interconnected predictors, Elastic Net and also Ridge logistic regression showed good performance compared to their competitors, while other methods did not lead to substantial improvements or even performed worse than standard logistic regression. Our work demonstrates how simulation studies that mimic relevant features of a specific data set can support the choice of a good modeling strategy.     

Results of a “GWAS Plus:” General Cognitive Ability Is Substantially Heritable and Massively Polygenic

We carried out a genome-wide association study (GWAS) for general cognitive ability (GCA) plus three other analyses of GWAS data that aggregate the effects of multiple single-nucleotide polymorphisms (SNPs) in various ways. Our multigenerational sample comprised 7,100 Caucasian participants, drawn from two longitudinal family studies, who had been assessed with an age-appropriate IQ test and had provided DNA samples passing quality screens. We conducted the GWAS across ∼2.5 million SNPs (both typed and imputed), using a generalized least-squares method appropriate for the different family structures present in our sample, and subsequently conducted gene-based association tests. We also conducted polygenic prediction analyses under five-fold cross-validation, using two different schemes of weighting SNPs. Using parametric bootstrapping, we assessed the performance of this prediction procedure under the null. Finally, we estimated the proportion of variance attributable to all genotyped SNPs as random effects with software GCTA. The study is limited chiefly by its power to detect realistic single-SNP or single-gene effects, none of which reached genome-wide significance, though some genomic inflation was evident from the GWAS. Unit SNP weights performed about as well as least-squares regression weights under cross-validation, but the performance of both increased as more SNPs were included in calculating the polygenic score. Estimates from GCTA were 35% of phenotypic variance at the recommended biological-relatedness ceiling. Taken together, our results concur with other recent studies: they support a substantial heritability of GCA, arising from a very large number of causal SNPs, each of very small effect. We place our study in the context of the literature–both contemporary and historical–and provide accessible explication of our statistical methods.