Evaluating the effects of rater and subject factors on measures of association

Large‐scale agreement studies are becoming increasingly common in medical settings to gain better insight into discrepancies often observed between experts' classifications. Ordered categorical scales are routinely used to classify subjects' disease and health conditions. Summary measures such as Cohen's weighted kappa are popular approaches for reporting levels of association for pairs of raters' ordinal classifications. However, in large‐scale studies with many raters, assessing levels of association can be challenging due to dependencies between many raters each grading the same sample of subjects' results and the ordinal nature of the ratings. Further complexities arise when the focus of a study is to examine the impact of rater and subject characteristics on levels of association. In this paper, we describe a flexible approach based upon the class of generalized linear mixed models to assess the influence of rater and subject factors on association between many raters' ordinal classifications. We propose novel model‐based measures for large‐scale studies to provide simple summaries of association similar to Cohen's weighted kappa while avoiding prevalence and marginal distribution issues that Cohen's weighted kappa is susceptible to. The proposed summary measures can be used to compare association between subgroups of subjects or raters. We demonstrate the use of hypothesis tests to formally determine if rater and subject factors have a significant influence on association, and describe approaches for evaluating the goodness‐of‐fit of the proposed model. The performance of the proposed approach is explored through extensive simulation studies and is applied to a recent large‐scale cancer breast cancer screening study.     

Modeling kappa for measuring dependent categorical agreement data

A method for analysing dependent agreement data with categorical responses is proposed. A generalized estimating equation approach is developed with two sets of equations. The first set models the marginal distribution of categorical ratings, and the second set models the pairwise association of ratings with the kappa coefficient (kappa) as a metric. Covariates can be incorporated into both sets of equations. This approach is compared with a latent variable model that assumes an underlying multivariate normal distribution in which the intraclass correlation coefficient is used as a measure of association. Examples are from a cervical ectopy study and the National Heart, Lung, and Blood Institute Veteran Twin Study.     

Joint species distribution modelling for spatio‐temporal occurrence and ordinal abundance data

Aim

Species distribution models are important tools used to study the distribution and abundance of organisms relative to abiotic variables. Dynamic local interactions among species in a community can affect abundance. The abundance of a single species may not be at equilibrium with the environment for spreading invasive species and species that are range shifting because of climate change. Innovation : We develop methods for incorporating temporal processes into a spatial joint species distribution model for presence/absence and ordinal abundance data. We model non‐equilibrium conditions via a temporal random effect and temporal dynamics with a vector‐autoregressive process allowing for intra‐ and interspecific dependence between co‐occurring species. The autoregressive term captures how the abundance of each species can enhance or inhibit its own subsequent abundance or the subsequent abundance of other species in the community and is well suited for a ‘community modules’ approach of strongly interacting species within a food web. R code is provided for fitting multispecies models within a Bayesian framework for ordinal data with any number of locations, time points, covariates and ordinal categories.

Main conclusions

We model ordinal abundance data of two invasive insects (hemlock woolly adelgid and elongate hemlock scale) that share a host tree and were undergoing northwards range expansion in the eastern U.S.A. during the period 1997–2011. Accounting for range expansion and high inter‐annual variability in abundance led to improved estimation of the species–environment relationships. We would have erroneously concluded that winter temperatures did not affect scale abundance had we not accounted for the range expansion of scale. The autoregressive component revealed weak evidence for commensalism, in which adelgid may have predisposed hemlock stands for subsequent infestation by scale. Residual spatial dependence indicated that an unmeasured variable additionally affected scale abundance. Our robust modelling approach could provide similar insights for other community modules of co‐occurring species.     

Power analysis for trend in ordinal cover classes: implications for long‐term vegetation monitoring

Question: We provide a method to calculate the power of ordinal regression models for detecting temporal trends in plant abundance measured as ordinal cover classes. Does power depend on the shape of the unobserved (latent) distribution of percentage cover? How do cover class schemes that differ in the number of categories affect power? Methods: We simulated cover class data by “cutting‐up” a continuous logit‐beta distributed variable using 7‐point and 15‐point cover classification schemes. We used Monte Carlo simulation to estimate power for detecting trends with two ordinal models, proportional odds logistic regression (POM) and logistic regression with cover classes re‐binned into two categories, a model we term an assessment point model (APM). We include a model fit to the logit‐transformed percentage cover data for comparison, which is a latent model. Results: The POM had equal or higher power compared to the APM and latent model, but power varied in complex ways as a function of the assumed latent beta distribution. We discovered that if the latent distribution is skewed, a cover class scheme with more categories might yield higher power to detect trend. Conclusions: Our power analysis method maintains the connection between the observed ordinal cover classes and the unmeasured (latent) percentage cover variable, allowing for a biologically meaningful trend to be defined on the percentage cover scale. Both the shape of the latent beta distribution and the alternative hypothesis should be considered carefully when determining sample size requirements for long‐term vegetation monitoring using cover class measurements.     

Cross‐scale predictive modeling of CHO cell culture growth and metabolites using Raman spectroscopy and multivariate analysis

Multi‐component, multi‐scale Raman spectroscopy modeling results from a monoclonal antibody producing CHO cell culture process including data from two development scales (3 L, 200 L) and a clinical manufacturing scale environment (2,000 L) are presented. Multivariate analysis principles are a critical component to partial least squares (PLS) modeling but can quickly turn into an overly iterative process, thus a simplified protocol is proposed for addressing necessary steps including spectral preprocessing, spectral region selection, and outlier removal to create models exclusively from cell culture process data without the inclusion of spectral data from chemically defined nutrient solutions or targeted component spiking studies. An array of single‐scale and combination‐scale modeling iterations were generated to evaluate technology capabilities and model scalability. Analysis of prediction errors across models suggests that glucose, lactate, and osmolality are well modeled. Model strength was confirmed via predictive validation and by examining performance similarity across single‐scale and combination‐scale models. Additionally, accurate predictive models were attained in most cases for viable cell density and total cell density; however, these components exhibited some scale‐dependencies that hindered model quality in cross‐scale predictions where only development data was used in calibration. Glutamate and ammonium models were also able to achieve accurate predictions in most cases. However, there are differences in the absolute concentration ranges of these components across the datasets of individual bioreactor scales. Thus, glutamate and ammonium PLS models were forced to extrapolate in cases where models were derived from small scale data only but used in cross‐scale applications predicting against manufacturing scale batches. © 2014 American Institute of Chemical Engineers Biotechnol. Prog., 31:566–577, 2015     

Measuring intrarater association between correlated ordinal ratings

Variability between raters' ordinal scores is commonly observed in imaging tests, leading to uncertainty in the diagnostic process. In breast cancer screening, a radiologist visually interprets mammograms and MRIs, while skin diseases, Alzheimer's disease, and psychiatric conditions are graded based on clinical judgment. Consequently, studies are often conducted in clinical settings to investigate whether a new training tool can improve the interpretive performance of raters. In such studies, a large group of experts each classify a set of patients' test results on two separate occasions, before and after some form of training with the goal of assessing the impact of training on experts' paired ratings. However, due to the correlated nature of the ordinal ratings, few statistical approaches are available to measure association between raters' paired scores. Existing measures are restricted to assessing association at just one time point for a single screening test. We propose here a novel paired kappa to provide a summary measure of association between many raters' paired ordinal assessments of patients' test results before versus after rater training. Intrarater association also provides valuable insight into the consistency of ratings when raters view a patient's test results on two occasions with no intervention undertaken between viewings. In contrast to existing correlated measures, the proposed kappa is a measure that provides an overall evaluation of the association among multiple raters' scores from two time points and is robust to the underlying disease prevalence. We implement our proposed approach in two recent breast-imaging studies and conduct extensive simulation studies to evaluate properties and performance of our summary measure of association.     

Calculating Ordinal Regression Models in SAS and S‐Plus

Although a number of regression models for ordinal responses have been proposed, these models are not widely known and applied in epidemiology and biomedical research. Overviews of these models are either highly technical or consider only a small part of this class of models so that it is difficult to understand the features of the models and to recognize important relations between them. In this paper we give an overview of logistic regression models for ordinal data based upon cumulative and conditional probabilities. We show how the most popular ordinal regression models, namely the proportional odds model and the continuation ratio model, are embedded in the framework of generalized linear models. We describe the characteristics and interpretations of these models and show how the calculations can be performed by means of SAS and S‐Plus. We illustrate and compare the methods by applying them to data of a study investigating the effect of several risk factors on diabetic retinopathy. A special aspect is the violation of the usual assumption of equal slopes which makes the correct application of standard models impossible. We show how to use extensions of the standard models to work adequately with this situation.     

Weighted least-squares approach for comparing correlated kappa

In the medical sciences, studies are often designed to assess the agreement between different raters or different instruments. The kappa coefficient is a popular index of agreement for binary and categorical ratings. Here we focus on testing for the equality of two dependent kappa coefficients. We use the weighted least-squares (WLS) approach of Koch et al. (1977, Biometrics 33, 133-158) to take into account the correlation between the estimated kappa statistics. We demonstrate how the SAS PROC CATMOD can be used to test for the equality of dependent Cohen's kappa coefficients and dependent intraclass kappa coefficients with nominal categorical ratings. We also test for the equality of dependent Cohen's kappa and dependent weighted kappa with ordinal ratings. The major advantage of the WLS approach is that it allows the data analyst a way of testing dependent kappa with popular SAS software. The WLS approach can handle any number of categories. Analyses of three biomedical studies are used for illustration.     

Regression modeling of ordinal data with nonzero baselines

This paper develops regression models for ordinal data with nonzero control response probabilities. The models are especially useful in dose-response studies where the spontaneous or natural response rate is nonnegligible and the dosage is logarithmic. These models generalize Abbott's formula, which has been commonly used to model binary data with nonzero background observations. We describe a biologically plausible latent structure and develop an EM algorithm for fitting the models. The EM algorithm can be implemented using standard software for ordinal regression. A toxicology data set where the proposed model fits the data but a more conventional model fails is used to illustrate the methodology.     

Marginal analysis of longitudinal ordinal data with misclassification in both response and covariates

Marginal methods have been widely used for the analysis of longitudinal ordinal and categorical data. These models do not require full parametric assumptions on the joint distribution of repeated response measurements but only specify the marginal or even association structures. However, inference results obtained from these methods often incur serious bias when variables are subject to error. In this paper, we tackle the problem that misclassification exists in both response and categorical covariate variables. We develop a marginal method for misclassification adjustment, which utilizes second‐order estimating functions and a functional modeling approach, and can yield consistent estimates and valid inference for mean and association parameters. We propose a two‐stage estimation approach for cases in which validation data are available. Our simulation studies show good performance of the proposed method under a variety of settings. Although the proposed method is phrased to data with a longitudinal design, it also applies to correlated data arising from clustered and family studies, in which association parameters may be of scientific interest. The proposed method is applied to analyze a dataset from the Framingham Heart Study as an illustration.     

Ordinal response regression models in ecology

Abstract. Although ordinal data are not rare in ecology, ecological studies have, until now, seriously neglected the use of specific ordinal regression models. Here, we present three models – the Proportional Odds, the Continuation Ratio and the Stereotype models – that can be successfully applied to ordinal data. Their differences and respective fields of application are discussed. Finally, as an example of application, PO models are used to predict spatial abundance of plant species in a Geographical Information System. It shows that ordinal models give as good a result as binary logistic models for predicting presence‐absence, but are additionally able to predict abundance satisfactorily.     

Notes on Estimation of the General Odds Ratio and the General Risk Difference for Paired‐Sample Data

Under the matched‐pair design, this paper discusses estimation of the general odds ratio OR_G for ordinal exposure in case‐control studies and the general risk difference RD_G for ordinal outcomes in cross‐sectional or cohort studies. To illustrate the practical usefulness of interval estimators of OR_G and RD_G developed here, this paper uses the data from a case‐control study investigating the effect of the number of beverages drunk at “burning hot” temperature on the risk of possessing esophageal cancer, and the data from a cross‐sectional study comparing the grade distributions of unaided distance vision between two eyes. Finally, this paper notes that using the commonly‐used statistics related to odds ratio for dichotomous data by collapsing the ordinal exposure into two categories: the exposure versus the non‐exposure, tends to be less efficient than using the statistics related to OR_G proposed herein.     

Incorporating Inter‐item Correlations in Item Response Data Analysis

This paper concerns with the analysis of item response data, which are usually measured on a rating scale and are therefore ordinal. These study items tended to be highly inter‐correlated. Rasch models, which convert ordinal categorical scales into linear measurements, are widely used in ordinal data analysis. In this paper, we improve the current methodology in order to incorporate inter‐item correlations. We have advocated the latent variable approach for this purpose, in combination with generalized estimating equations to estimate the Rasch model parameters. The data on a study of families of lung cancer patients demonstrate the utility of our methods.     

Strategies for genetic mapping of categorical traits

The search for efficient and powerful statistical methods and optimal mapping strategies for categorical traits under various experimental designs continues to be one of the main tasks in genetic mapping studies. Methodologies for genetic mapping of categorical traits can generally be classified into two groups, linear and non-linear models. We develop a method based on a threshold model, termed mixture threshold model to handle ordinal (or binary) data from multiple families. Monte Carlo simulations are done to compare its statistical efficiencies and properties of the proposed non-linear model with a linear model for genetic mapping of categorical traits using multiple families. The mixture threshold model has notably higher statistical power than linear models. There may be an optimal sampling strategy (family size vs number of families) in which genetic mapping reaches its maximal power and minimal estimation errors. A single large-sibship family does not necessarily produce the maximal power for detection of quantitative trait loci (QTL) due to genetic sampling of QTL alleles. The QTL allelic model has a marked impact on efficiency of genetic mapping of categorical traits in terms of statistical power and QTL parameter estimation. Compared with a fixed number of QTL alleles (two or four), the model with an infinite number of QTL alleles and normally distributed allelic effects results in loss of statistical power. The results imply that inbred designs (e.g. F₂ or four-way crosses) with a few QTL alleles segregating or reducing number of QTL alleles (e.g. by selection) in outbred populations are desirable in genetic mapping of categorical traits using data from multiple families. This revised version was published online in July 2006 with corrections to the Cover Date.     

Efficient approximate k‐fold and leave‐one‐out cross‐validation for ridge regression

In model building and model evaluation, cross‐validation is a frequently used resampling method. Unfortunately, this method can be quite time consuming. In this article, we discuss an approximation method that is much faster and can be used in generalized linear models and Cox’ proportional hazards model with a ridge penalty term. Our approximation method is based on a Taylor expansion around the estimate of the full model. In this way, all cross‐validated estimates are approximated without refitting the model. The tuning parameter can now be chosen based on these approximations and can be optimized in less time. The method is most accurate when approximating leave‐one‐out cross‐validation results for large data sets which is originally the most computationally demanding situation. In order to demonstrate the method's performance, it will be applied to several microarray data sets. An R package penalized, which implements the method, is available on CRAN.     

Five years of Homo floresiensis

Researchers have hypothesized that the degree to which an individual's actual behavior approximates the culturally valued lifestyle encoded in the dominant cultural model has consequences for physical and mental health. We contribute to this line of research by analyzing data from a longitudinal study composed of five annual surveys (2002–2006 inclusive) from 791 adults in one society of foragers–farmers in the Bolivian Amazon, the Tsimane'. We estimate the association between a standard measure of individual achievement of the cultural model, cultural consonance, and three indicators of body morphology. Drawing on research suggesting that in societies in the early stages of economic development an increase in socioeconomic status is associated with an increase in mean body mass, we expect to find a positive association between cultural consonance and three anthropometric measures. We found the expected positive association between cultural consonance and anthropometric measures—especially for men—only when using ordinary least square (OLS) regression models, but not when using fixed‐effects regression models. The real magnitude of the association was low. The comparison of estimates from OLS and fixed‐effect regression models suggests that previous findings on the effects of cultural consonance on body morphology using cross‐sectional data should be read with caution because the association might be largely explained by fixed characteristics of individuals not accounted in OLS models. Am J Phys Anthropol 143:167–174, 2010. © 2010 Wiley‐Liss, Inc.     

An isoeffect approach to the study of combined effects of mixed radiations--the nonparametric analysis of in vivo data

The combined effects of mixed radiations can be examined using a system of simple isoeffect relations which are derived from a recent analysis of in vitro results obtained for a variety of radiation mixtures. Similar isoeffect analysis methods have been used for over two decades in studies of the combined action of toxic agents such as drugs and antibiotics. Because of the isoeffect approach, the method is particularly useful for the analysis of ordinal data for which conventional models that are based on parametric dose-effect relations may not be suitable. This is illustrated by applying the method to the analysis of a set of recently published in vivo data using the mouse foot skin reaction system for mixtures of neutrons and X rays. The good agreement between this method and the ordinal data also helps to provide further experimental support for the existence of a class of radiobiological data for which the simple isoeffect relations are valid.     

Bayesian multilocus association mapping on ordinal and censored traits and its application to the analysis of genetic variation among Oryza sativa L. germplasms

Association mapping can be a powerful tool for detecting quantitative trait loci (QTLs) without requiring line-crossing experiments. We previously proposed a Bayesian approach for simultaneously mapping multiple QTLs by a regression method that directly incorporates estimates of the population structure. In the present study, we extended our method to analyze ordinal and censored traits, since both types of traits are common in the evaluation of germplasm collections. Ordinal-probit and tobit models were employed to analyze ordinal and censored traits, respectively. In both models, we postulated the existence of a latent continuous variable associated with the observable data, and we used a Markov-chain Monte Carlo algorithm to sample the latent variable and determine the model parameters. We evaluated the efficiency of our approach by using simulated- and real-trait analyses of a rice germplasm collection. Simulation analyses based on real marker data showed that our models could reduce both false-positive and false-negative rates in detecting QTLs to reasonable levels. Simulation analyses based on highly polymorphic marker data, which were generated by coalescent simulations, showed that our models could be applied to genotype data based on highly polymorphic marker systems, like simple sequence repeats. For the real traits, we analyzed heading date as a censored trait and amylose content and the shape of milled rice grains as ordinal traits. We found significant markers that may be linked to previously reported QTLs. Our approach will be useful for whole-genome association mapping of ordinal and censored traits in rice germplasm collections.     

Use of ordinal categorical variables in skeletal assessment of sex from the cranium

In anthropological studies, visual indicators of sex are traditionally scored on an ordinal categorical scale. Logistic and probit regression models are commonly used statistical tools for the analysis of ordinal categorical data. These models provide unbiased estimates of the posterior probabilities of sex conditional on observed indicators, but they do so only under certain conditions. We suggest a more general method for sexing using a multivariate cumulative probit model and examine both single indicator and multivariate indicator models on a sample of 138 crania from a Late Mississippian site in middle Tennessee. The crania were scored for five common sex indicators: superciliary arch form, chin form, size of mastoid process, shape of the supraorbital margin, and nuchal cresting. Independent assessment of sex for each individual is based on pubic indicators. The traditional logistic regressions are cumbersome because of limitations imposed by missing data. The logistic regression correctly classified 66/74 males and 46/64 females, with an overall correct classification of 81%. The cumulative probit model classified 64/74 males correctly and 51/64 females correctly for an overall correct classification rate of 83%. Finally, we apply parameters estimated from the logit and probit models to find posterior probabilities of sex assignment for 296 additional crania for which pubic indicators were absent or ambiguous. Am J Phys Anthropol 107:97–112, 1998. © 1998 Wiley-Liss, Inc.     

Incorporating evolutionary processes into conservation planning using species distribution data: a case study with the western Mediterranean herpetofauna

Aim To incorporate evolutionary processes into conservation planning using species distribution patterns and environmental gradients as surrogates for genetic diversity. Location Western Mediterranean Basin. Methods Distributions of 154 herpetological species were predicted using maximum entropy models, and groups of significantly co‐occurring species (biotic elements) were identified. Environmental gradients were characterized for the complete area and for the area covered by each biotic element, by performing a principal component analysis on the data matrix composed of nine environmental variables. The first two principal component analysis axes were classified into four categories each, and those categories were combined with each other resulting in an environmental classification with 16 categories. To identify priority conservation areas, biotic elements and environmental categories were used as surrogates for the neutral and adaptive components of genetic diversity, respectively. Priority areas for conservation were identified under three scenarios: (1) setting targets for species only; (2) setting targets for species and for each environmental category of the overall area; and (3) setting targets for each species and for each environmental category within each biotic element. Results Nine biotic elements were identified – four for the amphibians and five for the reptiles. Priority areas identified in the three scenarios were similar in terms of amount of area selected, but exhibited low spatial agreement. Main conclusions Prioritization exercises that integrate surrogates for evolutionary processes can deliver spatial priorities that are fairly different to those where only species representation is considered. While new methods are emerging to incorporate molecular data in conservation prioritization, it is unlikely to be enough data for enough taxa for this to be feasible in many regions. We develop an approach using surrogates for both the neutral and adaptive components of genetic diversity that may enhance biodiversity persistence and representation when molecular data are not available or geographically comprehensive.