Weighted least-squares approach for comparing correlated kappa

In the medical sciences, studies are often designed to assess the agreement between different raters or different instruments. The kappa coefficient is a popular index of agreement for binary and categorical ratings. Here we focus on testing for the equality of two dependent kappa coefficients. We use the weighted least-squares (WLS) approach of Koch et al. (1977, Biometrics 33, 133-158) to take into account the correlation between the estimated kappa statistics. We demonstrate how the SAS PROC CATMOD can be used to test for the equality of dependent Cohen's kappa coefficients and dependent intraclass kappa coefficients with nominal categorical ratings. We also test for the equality of dependent Cohen's kappa and dependent weighted kappa with ordinal ratings. The major advantage of the WLS approach is that it allows the data analyst a way of testing dependent kappa with popular SAS software. The WLS approach can handle any number of categories. Analyses of three biomedical studies are used for illustration.     

Evaluating the effects of rater and subject factors on measures of association

Large‐scale agreement studies are becoming increasingly common in medical settings to gain better insight into discrepancies often observed between experts' classifications. Ordered categorical scales are routinely used to classify subjects' disease and health conditions. Summary measures such as Cohen's weighted kappa are popular approaches for reporting levels of association for pairs of raters' ordinal classifications. However, in large‐scale studies with many raters, assessing levels of association can be challenging due to dependencies between many raters each grading the same sample of subjects' results and the ordinal nature of the ratings. Further complexities arise when the focus of a study is to examine the impact of rater and subject characteristics on levels of association. In this paper, we describe a flexible approach based upon the class of generalized linear mixed models to assess the influence of rater and subject factors on association between many raters' ordinal classifications. We propose novel model‐based measures for large‐scale studies to provide simple summaries of association similar to Cohen's weighted kappa while avoiding prevalence and marginal distribution issues that Cohen's weighted kappa is susceptible to. The proposed summary measures can be used to compare association between subgroups of subjects or raters. We demonstrate the use of hypothesis tests to formally determine if rater and subject factors have a significant influence on association, and describe approaches for evaluating the goodness‐of‐fit of the proposed model. The performance of the proposed approach is explored through extensive simulation studies and is applied to a recent large‐scale cancer breast cancer screening study.     

Analysis of ordered categorical data: two score-independent approaches

SUMMARY: A trend test is often employed to analyze ordered categorical data, in which a set of increasing scores is assigned a priori. There is a drawback in this approach, because how to choose a set of scores is not clear. There have been debates on which scores should be used (e.g., Graubard and Korn, 1987, Biometrics 43, 471-476; Ivanova and Berger, 2001, Biometrics 57, 567-570; Senn, 2007, Biometrics 63, 296-298). Conflicting conclusions are often obtained with different sets of scores. Two approaches, which have been applied to genetic case-control studies, are appealing for ordered categorical data, because they take into account the natural order in the data, are score independent, and not contingent on asymptotic theory. These two approaches are applied to a prospective study for detecting association between maternal drinking and congenital malformations.     

Modeling concordance correlation via GEE to evaluate reproducibility

Clinical studies are often concerned with assessing whether different raters/methods produce similar values for measuring a quantitative variable. Use of the concordance correlation coefficient as a measure of reproducibility has gained popularity in practice since its introduction by Lin (1989, Biometrics 45, 255-268). Lin's method is applicable for studies evaluating two raters/two methods without replications. Chinchilli et al. (1996, Biometrics 52, 341-353) extended Lin's approach to repeated measures designs by using a weighted concordance correlation coefficient. However, the existing methods cannot easily accommodate covariate adjustment, especially when one needs to model agreement. In this article, we propose a generalized estimating equations (GEE) approach to model the concordance correlation coefficient via three sets of estimating equations. The proposed approach is flexible in that (1) it can accommodate more than two correlated readings and test for the equality of dependent concordant correlation estimates; (2) it can incorporate covariates predictive of the marginal distribution; (3) it can be used to identify covariates predictive of concordance correlation; and (4) it requires minimal distribution assumptions. A simulation study is conducted to evaluate the asymptotic properties of the proposed approach. The method is illustrated with data from two biomedical studies.     

Marginal analysis of longitudinal ordinal data with misclassification in both response and covariates

Marginal methods have been widely used for the analysis of longitudinal ordinal and categorical data. These models do not require full parametric assumptions on the joint distribution of repeated response measurements but only specify the marginal or even association structures. However, inference results obtained from these methods often incur serious bias when variables are subject to error. In this paper, we tackle the problem that misclassification exists in both response and categorical covariate variables. We develop a marginal method for misclassification adjustment, which utilizes second‐order estimating functions and a functional modeling approach, and can yield consistent estimates and valid inference for mean and association parameters. We propose a two‐stage estimation approach for cases in which validation data are available. Our simulation studies show good performance of the proposed method under a variety of settings. Although the proposed method is phrased to data with a longitudinal design, it also applies to correlated data arising from clustered and family studies, in which association parameters may be of scientific interest. The proposed method is applied to analyze a dataset from the Framingham Heart Study as an illustration.     

Association models for a multivariate binary response

Models for a multivariate binary response are parameterized by univariate marginal probabilities and dependence ratios of all orders. The w-order dependence ratio is the joint success probability of w binary responses divided by the joint success probability assuming independence. This parameterization supports likelihood-based inference for both regression parameters, relating marginal probabilities to explanatory variables, and association model parameters, relating dependence ratios to simple and meaningful mechanisms. Five types of association models are proposed, where responses are (1) independent given a necessary factor for the possibility of a success, (2) independent given a latent binary factor, (3) independent given a latent beta distributed variable, (4) follow a Markov chain, and (5) follow one of two first-order Markov chains depending on the realization of a binary latent factor. These models are illustrated by reanalyzing three data sets, foremost a set of binary time series on auranofin therapy against arthritis. Likelihood-based approaches are contrasted with approaches based on generalized estimating equations. Association models specified by dependence ratios are contrasted with other models for a multivariate binary response that are specified by odds ratios or correlation coefficients.     

Measuring intrarater association between correlated ordinal ratings

Variability between raters' ordinal scores is commonly observed in imaging tests, leading to uncertainty in the diagnostic process. In breast cancer screening, a radiologist visually interprets mammograms and MRIs, while skin diseases, Alzheimer's disease, and psychiatric conditions are graded based on clinical judgment. Consequently, studies are often conducted in clinical settings to investigate whether a new training tool can improve the interpretive performance of raters. In such studies, a large group of experts each classify a set of patients' test results on two separate occasions, before and after some form of training with the goal of assessing the impact of training on experts' paired ratings. However, due to the correlated nature of the ordinal ratings, few statistical approaches are available to measure association between raters' paired scores. Existing measures are restricted to assessing association at just one time point for a single screening test. We propose here a novel paired kappa to provide a summary measure of association between many raters' paired ordinal assessments of patients' test results before versus after rater training. Intrarater association also provides valuable insight into the consistency of ratings when raters view a patient's test results on two occasions with no intervention undertaken between viewings. In contrast to existing correlated measures, the proposed kappa is a measure that provides an overall evaluation of the association among multiple raters' scores from two time points and is robust to the underlying disease prevalence. We implement our proposed approach in two recent breast-imaging studies and conduct extensive simulation studies to evaluate properties and performance of our summary measure of association.     

Median regression models for longitudinal data with dropouts

Summary .  Recently, median regression models have received increasing attention. When continuous responses follow a distribution that is quite different from a normal distribution, usual mean regression models may fail to produce efficient estimators whereas median regression models may perform satisfactorily. In this article, we discuss using median regression models to deal with longitudinal data with dropouts. Weighted estimating equations are proposed to estimate the median regression parameters for incomplete longitudinal data, where the weights are determined by modeling the dropout process. Consistency and the asymptotic distribution of the resultant estimators are established. The proposed method is used to analyze a longitudinal data set arising from a controlled trial of HIV disease ( Volberding et al., 1990 , The New England Journal of Medicine 322, 941–949). Simulation studies are conducted to assess the performance of the proposed method under various situations. An extension to estimation of the association parameters is outlined.     

Inference for kappas for longitudinal study data: applications to sexual health research

Summary .   Analysis of instrument reliability and rater agreement is used in a wide range of behavioral, medical, psychosocial, and health-care-related research to assess psychometric properties of instruments, consensus in disease diagnoses, fidelity of psychosocial intervention, and accuracy of proxy outcomes. For categorical outcomes, Cohen's kappa is the most widely used index of agreement and reliability. In many modern-day applications, data are often clustered, making inference difficult to perform using existing methods. In addition, as longitudinal study designs become increasingly popular, missing data have become a serious issue, and the lack of methods to systematically address this problem has hampered the progress of research in the aforementioned fields. In this article, we develop a novel approach based on a new class of kappa estimates to tackle the complexities involved in addressing missing data and other related issues arising from a general multirater and longitudinal data setting. The approach is illustrated with real data in sexual health research.     

Analyzing k (2 x 2) tables under cluster sampling

In many data analytic applications, such as ophthalmologic, longitudinal, or periodontal studies, multiple observations are recorded over several sites (or timepoints) within the same subject, bringing about dependence between measurements. This correlation, in turn, precludes the use of standard statistical methods that assume independence between outcome measurements. For example, the Mantel-Haenszel statistic, used to assess association between a binary outcome and a binary exposure while adjusting for a categorical covariate, does not follow the usual chi-squared distribution under the null hypothesis when there is correlation between observations. A modified Mantel-Haenszel procedure, which makes adjustment for dependence, is proposed. No particular correlation structure is assumed for responses within a cluster. This closed-form adjustment stems from Liang and Zeger's (1986, Biometrika 73, 13-22) generalized estimating equations approach for clustered data. The difference between this tabular (i.e., noniterative) technique and many earlier tabular methods is that the current method allows for consideration of site-specific exposure and covariate information. An example from a periodontal research study illustrates application of the method.     

Maximum likelihood methods for cure rate models with missing covariates

We propose maximum likelihood methods for parameter estimation for a novel class of semiparametric survival models with a cure fraction, in which the covariates are allowed to be missing. We allow the covariates to be either categorical or continuous and specify a parametric distribution for the covariates that is written as a sequence of one-dimensional conditional distributions. We propose a novel EM algorithm for maximum likelihood estimation and derive standard errors by using Louis's formula (Louis, 1982, Journal of the Royal Statistical Society, Series B 44, 226-233). Computational techniques using the Monte Carlo EM algorithm are discussed and implemented. A real data set involving a melanoma cancer clinical trial is examined in detail to demonstrate the methodology.     

A theoretical model to predict distribution of the fabric tensor and apparent density in cancellous bone

 The adaptation of cancellous bone to mechanical forces is well recognized. Theoretical models for predicting cancellous bone architecture have been developed and have mainly focused on the distribution of trabecular mass or the apparent density. The purpose of this study was to develop a theoretical model which can simultaneously predict the distribution of trabecular orthotropy/orientation, as represented by the fabric tensor, along with apparent density. Two sets of equations were derived under the assumption that cancellous bone is a biological self-optimizing material which tends to minimize strain energy. The first set of equations provide the relationship between the fabric tensor and stress tensor, and have been verified to be consistent with Wolff’s law of trabecular architecture, that is, the principal directions of the fabric tensor coincide with the principal stress trajectories. The second set of equations yield the apparent density from the stress tensor, which was shown to be identical to those obtained based on local optimization with strain energy density of true bone tissue as the objective function. These two sets of equations, together with elasticity field equations, provide a complete mathematical formulation for the adaptation of cancellous bone. Received: 25 February 1997/Revised version: 23 September 1997     

Testing for marginal independence between two categorical variables with multiple responses

Questions that ask respondents to "choose all that apply" from a set of items occur frequently in surveys. Categorical variables that summarize this type of survey data are called both pick any/c variables and multiple-response categorical variables. It is often of interest to test for independence between two categorical variables. When both categorical variables can have multiple responses, traditional Pearson chi-square tests for independence should not be used because of the within-subject dependence among responses. An intuitively constructed version of the Pearson statistic is proposed to perform the test using bootstrap procedures to approximate its sampling distribution. First- and second-order adjustments to the proposed statistic are given in order to use a chi-square distribution approximation. A Bonferroni adjustment is proposed to perform the test when the joint set of responses for individual subjects is unavailable. Simulations show that the bootstrap procedures hold the correct size more consistently than the other procedures.     

Transformations of covariates for longitudinal data

This paper develops a general approach for dealing with parametric transformations of covariates for longitudinal data, where the responses are modeled marginally and generalized estimating equations (GEEs) are used for estimation of regression parameters. We propose an iterative algorithm for obtaining regression and transformation parameters from estimating equations, utilizing existing software for GEE problems. The algorithmic technique is closely related to that used in the Box-Tidwell transformation in classical linear regression, but we develop it under the GEE setting and for more general transformation functions. We provide supporting theorems for consistency and asymptotic Normality of the estimates. Inference between two nested models is also considered. This methodology is applied to two data sets. One consists of pill dissolution data, the other is taken from the Pittsburgh Youth Study (PYS). The PYS is a prospective longitudinal study of the development of delinquency, substance use, and mental health in male youth. We use the model-based parametric approach to examine the association between alcohol use at an early stage of adolescent development and delinquency over the course of adolescence.     

Latent transition regression for mixed outcomes

Health status is a complex outcome, often characterized by multiple measures. When assessing changes in health status over time, multiple measures are typically collected longitudinally. Analytic challenges posed by these multivariate longitudinal data are further complicated when the outcomes are combinations of continuous, categorical, and count data. To address these challenges, we propose a fully Bayesian latent transition regression approach for jointly analyzing a mixture of longitudinal outcomes from any distribution. Health status is assumed to be a categorical latent variable, and the multiple outcomes are treated as surrogate measures of the latent health state, observed with error. Using this approach, both baseline latent health state prevalences and the probabilities of transitioning between the health states over time are modeled as functions of covariates. The observed outcomes are related to the latent health states through regression models that include subject-specific effects to account for residual correlation among repeated measures over time, and covariate effects to account for differential measurement of the latent health states. We illustrate our approach with data from a longitudinal study of back pain.     

Inference of S-system models of genetic networks by solving one-dimensional function optimization problems

Voit and Almeida have proposed the decoupling approach as a method for inferring the S-system models of genetic networks. The decoupling approach defines the inference of a genetic network as a problem requiring the solutions of sets of algebraic equations. The computation can be accomplished in a very short time, as the approach estimates S-system parameters without solving any of the differential equations. Yet the defined algebraic equations are non-linear, which sometimes prevents us from finding reasonable S-system parameters. In this study, we propose a new technique to overcome this drawback of the decoupling approach. This technique transforms the problem of solving each set of algebraic equations into a one-dimensional function optimization problem. The computation can still be accomplished in a relatively short time, as the problem is transformed by solving a linear programming problem. We confirm the effectiveness of the proposed approach through numerical experiments.     

Generalized estimating equations for ordinal categorical data: arbitrary patterns of missing responses and missingness in a key covariate

We propose methods for regression analysis of repeatedly measured ordinal categorical data when there is nonmonotone missingness in these responses and when a key covariate is missing depending on observables. The methods use ordinal regression models in conjunction with generalized estimating equations (GEEs). We extend the GEE methodology to accommodate arbitrary patterns of missingness in the responses when this missingness is independent of the unobserved responses. We further extend the methodology to provide correction for possible bias when missingness in knowledge of a key covariate may depend on observables. The approach is illustrated with the analysis of data from a study in diagnostic oncology in which multiple correlated receiver operating characteristic curves are estimated and corrected for possible verification bias when the true disease status is missing depending on observables.     

Testing the correlation for clustered categorical and censored discrete time-to-event data when covariates are measured without/with errors

In the analysis of clustered categorical data, it is of common interest to test for the correlation within clusters, and the heterogeneity across different clusters. We address this problem by proposing a class of score tests for the null hypothesis that the variance components are zero in random effects models, for clustered nominal and ordinal categorical responses. We extend the results to accommodate clustered censored discrete time-to-event data. We next consider such tests in the situation where covariates are measured with errors. We propose using the SIMEX method to construct the score tests for the null hypothesis that the variance components are zero. Key advantages of the proposed score tests are that they can be easily implemented by fitting standard polytomous regression models and discrete failure time models, and that they are robust in the sense that no assumptions need to be made regarding the distributions of the random effects and the unobserved covariates. The asymptotic properties of the proposed tests are studied. We illustrate these tests by analyzing two data sets and evaluate their performance with simulations.     

Regression Models for Clinical Trials with Correlated Categorical Responses

Regression models for correlated categorical data are presented in which the covariance is a function of measured effects. The regression and covariance parameters are estimated by extended least square methods. A numerical example of a clinical trial comparing two antiemetic treatment regimes for patients receiving chemotherapy is given to illustrate the approach.