Semiparametric analysis of zero-inflated count data

Medical and public health research often involve the analysis of count data that exhibit a substantially large proportion of zeros, such as the number of heart attacks and the number of days of missed primary activities in a given period. A zero-inflated Poisson regression model, which hypothesizes a two-point heterogeneity in the population characterized by a binary random effect, is generally used to model such data. Subjects are broadly categorized into the low-risk group leading to structural zero counts and high-risk (or normal) group so that the counts can be modeled by a Poisson regression model. The main aim is to identify the explanatory variables that have significant effects on (i) the probability that the subject is from the low-risk group by means of a logistic regression formulation; and (ii) the magnitude of the counts, given that the subject is from the high-risk group by means of a Poisson regression where the effects of the covariates are assumed to be linearly related to the natural logarithm of the mean of the counts. In this article we consider a semiparametric zero-inflated Poisson regression model that postulates a possibly nonlinear relationship between the natural logarithm of the mean of the counts and a particular covariate. A sieve maximum likelihood estimation method is proposed. Asymptotic properties of the proposed sieve maximum likelihood estimators are discussed. Under some mild conditions, the estimators are shown to be asymptotically efficient and normally distributed. Simulation studies were carried out to investigate the performance of the proposed method. For illustration purpose, the method is applied to a data set from a public health survey conducted in Indonesia where the variable of interest is the number of days of missed primary activities due to illness in a 4-week period.     

A comparison between Poisson and zero-inflated Poisson regression models with an application to number of black spots in Corriedale sheep

Dark spots in the fleece area are often associated with dark fibres in wool, which limits its competitiveness with other textile fibres. Field data from a sheep experiment in Uruguay revealed an excess number of zeros for dark spots. We compared the performance of four Poisson and zero-inflated Poisson (ZIP) models under four simulation scenarios. All models performed reasonably well under the same scenario for which the data were simulated. The deviance information criterion favoured a Poisson model with residual, while the ZIP model with a residual gave estimates closer to their true values under all simulation scenarios. Both Poisson and ZIP models with an error term at the regression level performed better than their counterparts without such an error. Field data from Corriedale sheep were analysed with Poisson and ZIP models with residuals. Parameter estimates were similar for both models. Although the posterior distribution of the sire variance was skewed due to a small number of rams in the dataset, the median of this variance suggested a scope for genetic selection. The main environmental factor was the age of the sheep at shearing. In summary, age related processes seem to drive the number of dark spots in this breed of sheep.     

Bootstrap tests for overdispersion in a zero-inflated Poisson regression model

Ridout, Hinde, and Demétrio (2001, Biometrics 57, 219-223) derived a score test for testing a zero-inflated Poisson (ZIP) regression model against zero-inflated negative binomial (ZINB) alternatives. They mentioned that the score test using the normal approximation might underestimate the nominal significance level possibly for small sample cases. To remedy this problem, a parametric bootstrap method is proposed. It is shown that the bootstrap method keeps the significance level close to the nominal one and has greater power uniformly than the existing normal approximation for testing the hypothesis.     

Zero-Inflated Poisson Models and C.A.MAN: A Tutorial Collection of Evidence

Analysis of count data is required in many areas of biometric interest. Often the simple Poisson distribution is not appropriate, since an extra-number of zero counts occur in the count data. Some current approaches for the problem at hand are reviewed. It will be argued that these situations can often be easily modeled using the zero-inflated Poisson distribution. A variety of applications are considered in which this occurs. Possibilities are outlined on how the validity of the zero-inflated Poisson can be validated including a comparison with the nonparametric Poisson mixture maximum likelihood estimator.     

Smooth tests for the zero-inflated poisson distribution

In this article we construct three smooth goodness-of-fit tests for testing for the zero-inflated Poisson (ZIP) distribution against general smooth alternatives in the sense of Neyman. We apply our tests to a data set previously claimed to be ZIP distributed, and show that the ZIP is not a good model to describe the data. At rejection of the null hypothesis of ZIP, the individual components of the test statistic, which are directly related to interpretable parameters in a smooth model, may be used to gain insight into an alternative distribution.     

A quantile count model of water depth constraints on Cape Sable seaside sparrows

1. A quantile regression model for counts of breeding Cape Sable seaside sparrows Ammodramus maritimus mirabilis (L.) as a function of water depth and previous year abundance was developed based on extensive surveys, 1992-2005, in the Florida Everglades. The quantile count model extends linear quantile regression methods to discrete response variables, providing a flexible alternative to discrete parametric distributional models, e.g. Poisson, negative binomial and their zero-inflated counterparts. 2. Estimates from our multiplicative model demonstrated that negative effects of increasing water depth in breeding habitat on sparrow numbers were dependent on recent occupation history. Upper 10th percentiles of counts (one to three sparrows) decreased with increasing water depth from 0 to 30 cm when sites were not occupied in previous years. However, upper 40th percentiles of counts (one to six sparrows) decreased with increasing water depth for sites occupied in previous years. 3. Greatest decreases (-50% to -83%) in upper quantiles of sparrow counts occurred as water depths increased from 0 to 15 cm when previous year counts were 1, but a small proportion of sites (5-10%) held at least one sparrow even as water depths increased to 20 or 30 cm. 4. A zero-inflated Poisson regression model provided estimates of conditional means that also decreased with increasing water depth but rates of change were lower and decreased with increasing previous year counts compared to the quantile count model. Quantiles computed for the zero-inflated Poisson model enhanced interpretation of this model but had greater lack-of-fit for water depths > 0 cm and previous year counts 1, conditions where the negative effect of water depths were readily apparent and fitted better with the quantile count model.     

Bagging GLM: Improved generalized linear model for the analysis of zero-inflated data

Species-occurrence data sets tend to contain a large proportion of zero values, i.e., absence values (zero-inflated). Statistical inference using such data sets is likely to be inefficient or lead to incorrect conclusions unless the data are treated carefully. In this study, we propose a new modeling method to overcome the problems caused by zero-inflated data sets that involves a regression model and a machine-learning technique. We combined a generalized liner model (GLM), which is widely used in ecology, and bootstrap aggregation (bagging), a machine-learning technique. We established distribution models of Vincetoxicum pycnostelma (a vascular plant) and Ninox scutulata (an owl), both of which are endangered and have zero-inflated distribution patterns, using our new method and traditional GLM and compared model performances. At the same time we modeled four theoretical data sets that contained different ratios of presence/absence values using new and traditional methods and also compared model performances. For distribution models, our new method showed good performance compared to traditional GLMs. After bagging, area under the curve (AUC) values were almost the same as with traditional methods, but sensitivity values were higher. Additionally, our new method showed high sensitivity values compared to the traditional GLM when modeling a theoretical data set containing a large proportion of zero values. These results indicate that our new method has high predictive ability with presence data when analyzing zero-inflated data sets. Generally, predicting presence data is more difficult than predicting absence data. Our new modeling method has potential for advancing species distribution modeling.     

The distribution of the pathogenic nematode Nematodirus battus in lambs is zero-inflated

Understanding the frequency distribution of parasites and parasite stages among hosts is essential for efficient experimental design and statistical analysis, and is also required for the development of sustainable methods of controlling infection. Nematodirus battus is one of the most important organisms that infect sheep but the distribution of parasites among hosts is unknown. An initial analysis indicated a high frequency of animals without N. battus and with zero egg counts, suggesting the possibility of a zero-inflated distribution. We developed a Bayesian analysis using Markov chain Monte Carlo methods to estimate the parameters of the zero-inflated negative binomial distribution. The analysis of 3000 simulated data sets indicated that this method out-performed the maximum likelihood procedure. Application of this technique to faecal egg counts from lambs in a commercial upland flock indicated that N. battus counts were indeed zero-inflated. Estimating the extent of zero-inflation is important for effective statistical analysis and for the accurate identification of genetically resistant animals.     

The k-ZIG: Flexible Modeling for Zero-Inflated Counts

Summary Many applications involve count data from a process that yields an excess number of zeros. Zero-inflated count models, in particular, zero-inflated Poisson (ZIP) and zero-inflated negative binomial (ZINB) models, along with Poisson hurdle models, are commonly used to address this problem. However, these models struggle to explain extreme incidence of zeros (say more than 80%), especially to find important covariates. In fact, the ZIP may struggle even when the proportion is not extreme. To redress this problem we propose the class of k-ZIG models. These models allow more flexible modeling of both the zero-inflation and the nonzero counts, allowing interplay between these two components. We develop the properties of this new class of models, including reparameterization to a natural link function. The models are straightforwardly fitted within a Bayesian framework. The methodology is illustrated with simulated data examples as well as a forest seedling dataset obtained from the USDA Forest Service's Forest Inventory and Analysis program.     

Bayesian interval mapping of count trait loci based on zero-inflated generalized Poisson regression model

Count phenotypes with excessive zeros are often observed in the biological world. Researchers have studied many statistical methods for mapping the quantitative trait loci (QTLs) of zero-inflated count phenotypes. However, most of the existing methods consist of finding the approximate positions of the QTLs on the chromosome by genome-wide scanning. Additionally, most of the existing methods use the EM algorithm for parameter estimation. In this paper, we propose a Bayesian interval mapping scheme of QTLs for zero-inflated count data. The method takes advantage of a zero-inflated generalized Poisson (ZIGP) regression model to study the influence of QTLs on the zero-inflated count phenotype. The MCMC algorithm is used to estimate the effects and position parameters of QTLs. We use the Haldane map function to realize the conversion between recombination rate and map distance. Monte Carlo simulations are conducted to test the applicability and advantage of the proposed method. The effects of QTLs on the formation of mouse cholesterol gallstones were demonstrated by analyzing an mouse data set.     

Modelling bivariate count series with excess zeros

Bivariate time series of counts with excess zeros relative to the Poisson process are common in many bioscience applications. Failure to account for the extra zeros in the analysis may result in biased parameter estimates and misleading inferences. A class of bivariate zero-inflated Poisson autoregression models is presented to accommodate the zero-inflation and the inherent serial dependency between successive observations. An autoregressive correlation structure is assumed in the random component of the compound regression model. Parameter estimation is achieved via an EM algorithm, by maximizing an appropriate log-likelihood function to obtain residual maximum likelihood estimates. The proposed method is applied to analyze a bivariate series from an occupational health study, in which the zero-inflated injury count events are classified as either musculoskeletal or non-musculoskeletal in nature. The approach enables the evaluation of the effectiveness of a participatory ergonomics intervention at the population level, in terms of reducing the overall incidence of lost-time injury and a simultaneous decline in the two mean injury rates.     

When no catches matter: Coping with zeros in environmental assessments

The environmental legislation of many countries increasingly requires the continuous monitoring of fish assemblages to evaluate the success of river and stream restorations. Predicting species–environment relationships on the basis of monitoring data is central in the evaluation of ecological integrity and planning of rehabilitation strategies. Monitoring data are, however, often plagued by a substantial proportion of zeros (no catch at single sampling points) which are caused by relevant ecological processes, but complicate the use of commonly used statistical methods. This study compares mere count regression models, mixture and hurdle models based on Poisson and negative binomial distribution and logistic regressions with respect to their ability to cope with large zero-inflated data sets obtained by point abundance sampling of young-of-the-year fish from three large German rivers. Only mixture and hurdle models based on negative binomial distribution could satisfactorily be fitted to the zero-inflated and overdispersed count data. The logistic regression models applied to transliterated catch data simplified the computational procedure and yielded qualitative similar results to the count regression models indicating that the use of more complex count data did not generally provide better predictions. Therefore, presence/absence sampling may be a suitable and less costly alternative to abundance surveys for identifying environmental factors which affect the spatial distribution of fish populations at least if information on subtly abundance fluctuations is not needed. Mixture or hurdle models are particularly worth the additional effort if it is reasonable to distinguish between those environmental factors influencing the occurrence probability and others affecting the abundance. All models showed low sensitivity to rare guilds pointing to the need for a further development of statistical models for rare species whose management is a matter of growing environmental concern.     

Zero-inflated Poisson and binomial regression with random effects: a case study

In a 1992 Technometrics paper, Lambert (1992, 34, 1-14) described zero-inflated Poisson (ZIP) regression, a class of models for count data with excess zeros. In a ZIP model, a count response variable is assumed to be distributed as a mixture of a Poisson(lambda) distribution and a distribution with point mass of one at zero, with mixing probability p. Both p and lambda are allowed to depend on covariates through canonical link generalized linear models. In this paper, we adapt Lambert's methodology to an upper bounded count situation, thereby obtaining a zero-inflated binomial (ZIB) model. In addition, we add to the flexibility of these fixed effects models by incorporating random effects so that, e.g., the within-subject correlation and between-subject heterogeneity typical of repeated measures data can be accommodated. We motivate, develop, and illustrate the methods described here with an example from horticulture, where both upper bounded count (binomial-type) and unbounded count (Poisson-type) data with excess zeros were collected in a repeated measures designed experiment.     

Bayesian semiparametric zero-inflated Poisson model for longitudinal count data

This paper presents new methods, using a Bayesian approach, for analyzing longitudinal count data with excess zeros and nonlinear effects of continuously valued covariates. In longitudinal count data there are many problems that can make the use of a zero-inflated Poisson (ZIP) model ineffective. These problems are unobserved heterogeneity and nonlinear effects of continuously valued covariates. Our proposed semiparametric model can simultaneously handle these problems in a unified framework. The framework accounts for heterogeneity by incorporating random effects and has two components. The parametric component of the model which deals with the linear effects of time invariant covariates and the non-parametric component which gives an arbitrary smooth function to model the effect of time or time-varying covariates on the logarithm of mean count. The proposed methods are illustrated by analyzing longitudinal count data on the assessment of an efficacy of pesticides in controlling the reproduction of whitefly.     

Zero-inflated generalized Poisson regression mixture model for mapping quantitative trait loci underlying count trait with many zeros

Phenotypes measured in counts are commonly observed in nature. Statistical methods for mapping quantitative trait loci (QTL) underlying count traits are documented in the literature. The majority of them assume that the count phenotype follows a Poisson distribution with appropriate techniques being applied to handle data dispersion. When a count trait has a genetic basis, “naturally occurring” zero status also reflects the underlying gene effects. Simply ignoring or miss-handling the zero data may lead to wrong QTL inference. In this article, we propose an interval mapping approach for mapping QTL underlying count phenotypes containing many zeros. The effects of QTLs on the zero-inflated count trait are modelled through the zero-inflated generalized Poisson regression mixture model, which can handle the zero inflation and Poisson dispersion in the same distribution. We implement the approach using the EM algorithm with the Newton-Raphson algorithm embedded in the M-step, and provide a genome-wide scan for testing and estimating the QTL effects. The performance of the proposed method is evaluated through extensive simulation studies. Extensions to composite and multiple interval mapping are discussed. The utility of the developed approach is illustrated through a mouse F₂ intercross data set. Significant QTLs are detected to control mouse cholesterol gallstone formation.     

Quasi Likelihood/Moment Method for Generalized and Restricted Generalized Poisson Regression Models and Its Application

This paper reviews the generalized Poisson regression model, the restricted generalized Poisson regression model and the mixed Poisson regression (negative binomial regression and Poisson inverse Gaussian regression) models which can be used for regression analysis of counts. The aim of this study is to demonstrate the quasi likelihood/moment method, which is used for estimation of the parameters of mixed Poisson regression models, also applicable to obtain the estimates of the parameters of the generalized Poisson regression and the restricted generalized Poisson regression models. Besides, at the end of this study an application related to this method for zoological data is given.     

Influences of size and sex on invasive species aggression and native species vulnerability: a case for modern regression techniques

Animal behaviour is of fundamental importance but is often overlooked in biological invasion research. A problem with such studies is that they may add pressure to already threatened species and subject vulnerable individuals to increased risk. One solution is to obtain the maximum possible information from the generated data using a variety of statistical techniques, instead of solely using simple versions of linear regression or generalized linear models as is customary. Here, we exemplify and compare the use of modern regression techniques which have very different conceptual backgrounds and aims (negative binomial models, zero-inflated regression, and expectile regression), and which have rarely been applied to behavioural data in biological invasion studies. We show that our data display overdispersion, which is frequent in ecological and behavioural data, and that conventional statistical methods such as Poisson generalized linear models are inadequate in this case. Expectile regression is similar to quantile regression and allows the estimation of functional relationships between variables for all portions of a probability distribution and is thus well suited for modelling boundaries in polygonal relationships or cases with heterogeneous variances which are frequent in behavioural data. We applied various statistical techniques to aggression in invasive mosquitofish, Gambusia holbrooki, and the concomitant vulnerability of native toothcarp, Aphanius iberus, in relation to individual size and sex. We found that medium sized male G. holbrooki carry out the majority of aggressive acts and that smaller and medium size A. iberus are most vulnerable. Of the regression techniques used, only negative binomial models and zero-inflated and expectile Poisson regressions revealed these relationships.