Measuring pairwise agreement among many observers

Weighted kappa is defined as a measure of pairwise inter observer agreement. A weighted intra class kappa coefficient is proposed to measure agreement on a particular response category. An interclass kappa coefficient is proposed for each pair of response categories. Simple estimation procedures are presented for the case where the observers judging one subject are not necessarily the same as those judging another subject. Large sample standard errors are derived and a numerical example is given.     

Evaluating the effects of rater and subject factors on measures of association

Large‐scale agreement studies are becoming increasingly common in medical settings to gain better insight into discrepancies often observed between experts' classifications. Ordered categorical scales are routinely used to classify subjects' disease and health conditions. Summary measures such as Cohen's weighted kappa are popular approaches for reporting levels of association for pairs of raters' ordinal classifications. However, in large‐scale studies with many raters, assessing levels of association can be challenging due to dependencies between many raters each grading the same sample of subjects' results and the ordinal nature of the ratings. Further complexities arise when the focus of a study is to examine the impact of rater and subject characteristics on levels of association. In this paper, we describe a flexible approach based upon the class of generalized linear mixed models to assess the influence of rater and subject factors on association between many raters' ordinal classifications. We propose novel model‐based measures for large‐scale studies to provide simple summaries of association similar to Cohen's weighted kappa while avoiding prevalence and marginal distribution issues that Cohen's weighted kappa is susceptible to. The proposed summary measures can be used to compare association between subgroups of subjects or raters. We demonstrate the use of hypothesis tests to formally determine if rater and subject factors have a significant influence on association, and describe approaches for evaluating the goodness‐of‐fit of the proposed model. The performance of the proposed approach is explored through extensive simulation studies and is applied to a recent large‐scale cancer breast cancer screening study.     

The kappa coefficient of agreement for multiple observers when the number of subjects is small

Published results on the use of the kappa coefficient of agreement have traditionally been concerned with situations where a large number of subjects is classified by a small group of raters. The coefficient is then used to assess the degree of agreement among the raters through hypothesis testing or confidence intervals. A modified kappa coefficient of agreement for multiple categories is proposed and a parameter-free distribution for testing null agreement is provided, for use when the number of raters is large relative to the number of categories and subjects. The large-sample distribution of kappa is shown to be normal in the nonnull case, and confidence intervals for kappa are provided. The results are extended to allow for an unequal number of raters per subject.     

Weighted least-squares approach for comparing correlated kappa

In the medical sciences, studies are often designed to assess the agreement between different raters or different instruments. The kappa coefficient is a popular index of agreement for binary and categorical ratings. Here we focus on testing for the equality of two dependent kappa coefficients. We use the weighted least-squares (WLS) approach of Koch et al. (1977, Biometrics 33, 133-158) to take into account the correlation between the estimated kappa statistics. We demonstrate how the SAS PROC CATMOD can be used to test for the equality of dependent Cohen's kappa coefficients and dependent intraclass kappa coefficients with nominal categorical ratings. We also test for the equality of dependent Cohen's kappa and dependent weighted kappa with ordinal ratings. The major advantage of the WLS approach is that it allows the data analyst a way of testing dependent kappa with popular SAS software. The WLS approach can handle any number of categories. Analyses of three biomedical studies are used for illustration.     

Measuring agreement of multivariate discrete survival times using a modified weighted kappa coefficient

Summary .  Assessing agreement is often of interest in clinical studies to evaluate the similarity of measurements produced by different raters or methods on the same subjects. We present a modified weighted kappa coefficient to measure agreement between bivariate discrete survival times. The proposed kappa coefficient accommodates censoring by redistributing the mass of censored observations within the grid where the unobserved events may potentially happen. A generalized modified weighted kappa is proposed for multivariate discrete survival times. We estimate the modified kappa coefficients nonparametrically through a multivariate survival function estimator. The asymptotic properties of the kappa estimators are established and the performance of the estimators are examined through simulation studies of bivariate and trivariate survival times. We illustrate the application of the modified kappa coefficient in the presence of censored observations with data from a prostate cancer study.     

Bayesian inference for kappa from single and multiple studies

Cohen's kappa coefficient is a widely popular measure for chance-corrected nominal scale agreement between two raters. This article describes Bayesian analysis for kappa that can be routinely implemented using Markov chain Monte Carlo (MCMC) methodology. We consider the case of m > or = 2 independent samples of measured agreement, where in each sample a given subject is rated by two rating protocols on a binary scale. A major focus here is on testing the homogeneity of the kappa coefficient across the different samples. The existing frequentist tests for this case assume exchangeability of rating protocols, whereas our proposed Bayesian test does not make any such assumption. Extensive simulation is carried out to compare the performances of the Bayesian and the frequentist tests. The developed methodology is illustrated using data from a clinical trial in ophthalmology.     

New statistical software for intralaboratory and interlaboratory quality control in clinical cytology. Validation in a simulation study on clinical samples

OBJECTIVE: To design a statistical software package to provide automated calculations of normal and weighted and 3 indices. STUDY DESIGN: Prompted by the lack of commonly available software to compute weighted kappa and the nonproportionate workload needed to calculate our 3 variability indices manually, the new statistical software package was designed. To demonstrate the performance of the new CONQUISTADOR software, a simulation study (both intralaboratory and interlaboratory) was designed using 5,000 clinical samples randomly selected from a data file of > or = 200,000 conventional Pap smears and programmed to become "analyzed" by 12 cytologists in 5 imaginary laboratories. RESULTS: A representative set of both complete and partial outputs provided by the software, in Excel format (Microsoft, Redmond, Washington, U.S.A.) are shown to illustrate the different functions of the program. In the interlaboratory mode, the software calculates accuracy indicators (sensitivity, specificity, positive and negative predictive value, and their 95% CI), which are not common features of regular statistical packages; kappa and weighted kappa; and their 95% CI (comparison of single laboratories to all laboratories and pairwise comparisons between single laboratories). The 3 diagnostic variability indices can be computed separately for all samples or for only the positive samples. In the intralaboratory mode, the software calculates the same indices for individual cytologists. CONCLUSION: The CONQUISTADOR statistical package has properties that are useful in monitoring cytologic laboratory quality in both intralaboratory and interlaboratory settings. The software will be distributed by the National Institute of Health, Rome, for the delivery costs only.     

Definitions and standardization of a new grading scheme for eyelid contour abnormalities after trichiasis surgery

Background

Clear definitions of outcomes following trichiasis surgery are critical for planning program evaluations and for identifying ways to improve trichiasis surgery. Eyelid contour abnormality is an important adverse outcome of surgery; however, no standard method has been described to categorize eyelid contour abnormalities.

Methodology/Principal Findings

A classification system for eyelid contour abnormalities following surgery for trachomatous trichiasis was developed. To determine whether the grading was reproducible using the classification system, six-week postoperative photographs were reviewed by two senior graders to characterize severity of contour abnormalities. Sample photographs defining each contour abnormality category were compiled and used to train four new graders. All six graders independently graded a Standardization Set of 75 eyelids, which included a roughly equal distribution across the severity scale, and weighted kappa scores were calculated. Two hundred forty six-week postoperative photographs from an ongoing clinical trial were randomly selected for evaluating agreement across graders. Two months after initial grading, one grader regraded a subset of the 240 photographs to measure longer-term intra-observer agreement. The weighted kappa for agreement between the two senior graders was 0.80 (95% CI: 0.71–0.89). Among the Standardization Set, agreement between the senior graders and the 4 new graders showed weighted kappa scores ranging from 0.60–0.80. Among 240 eyes comprising the clinical trial dataset, agreement ranged from weighted kappa 0.70–0.71. Longer-term intra-observer agreement was weighted kappa 0.86 (95% CI: 0.80–0.92).

Conclusions/Significance

The standard eyelid contour grading system we developed reproducibly delineates differing levels of contour abnormality. This grading system could be useful both for helping to evaluate trichiasis surgery outcomes in clinical trials and for evaluating trichiasis surgery programs.     

Interlaboratory quality control in gynecologic cytopathology using the novel CONQUISTADOR software. Interobserver reproducibility in the Latin American screening study

OBJECTIVE: The recently developed software (CONQUISTADOR), capable of computing all intralaboratory and interlaboratory quality control (QC) indicators, was used to evaluate the diagnostic agreement among 4 cytology laboratories participating in the LAMS Study. STUDY DESIGN: The study was an interlaboratory exchange of specially designed 5 slide sets, each comprising 20 (conventional cytology) slides. At the first step, 80 slides (with "clear-cut" cases) were divided into four sets (A, B, C, D) of 20 specimens, each including inadequate and negative cases as well as in different proportions of all diagnostic TBS 2001 categories. In the second round, a fifth set (E) of 20 slides ("difficult cases") was designed, with all diagnostic categories, ASC and AGC included. Common measures of reproducibility (kappa and weighted kappa), accuracy (SE, SP, PPV, NPV) and 3 indices of diagnostic variability were calculated for sets A-D and set E, separately. RESULTS: For the 5 slide sets together, the weighted kappa was 0.8 (95% CI 0.76-0.85), which is the lower limit of the "almost perfect" ranking of kappa statistics, indicating an excellent interlaboratory agreement. The interlaboratory reproducibility was lower only for the difficult set (E). Similarly, the sensitivity for set E (70.0%) was lower than that (92.1%) for sets A-D. The diagnostic variability indices were not substantially different between the difficult (set E) and clearcut (sets A-D) cases. CONCLUSION: High interlaboratory reproducibility was obtained for sets A-D ("clear-cut" cases), while more interlaboratory variation was evident in the difficult samples. The new CONQUISTADOR software is a valuable tool in calculating the indicators needed in this intralaboratory and interlaboratory.     

IgE myelomatosis. Presentation of a new case and summary of literature

This paper describes a case of IgE (kappa) myeloma in a 39 year old female patient who has been observed for a period of eight years to the present. The findings and the course of disease in this patient were compared with 18 other case reports published since the discovery of IgE in 1966. In contrast to myelomas of other immunoglobulin classes, patients with IgE myelomas are somewhat younger and anaemia and hyperproteinaemia are more pronounced. A plasma cell leukaemia is more frequent and the ratio of light chains has shifted in favour of the kappa chains.     

An Estimating Equations Approach for Modelling Kappa

Agreement between raters for binary outcome data is typically assessed using the kappa coefficient. There has been considerable recent work extending logistic regression to provide summary estimates of interrater agreement adjusted for covariates predictive of the marginal probability of classification by each rater. We propose an estimating equations approach which can also be used to identify covariates predictive of kappa. Models may include an arbitrary and variable number of raters per subject and yet do not require any stringent parametric assumptions. Examples used to illustrate this procedure include an investigation of factors affecting agreement between primary and proxy respondents from a case‐control study and a study of the effects of gender and zygosity on twin concordance for smoking history.     

Reliability and agreement of two smear reading scales for classification and monitoring of multidrug therapy in leprosy patients

Introduction. After the clinical diagnosis of leprosy, classification methods are necessary to define a treatment and prognosis of patients consistent with bacterial load. Bacteria are detected in skin smear, and bacterial load typically is established by the internationally used Ridley′s logarithmic scale, However, in Colombia an alternative semiquantitative scale is used. Objective. The interobserver reproducibility was established for the Ridley and Colombia scales, and the level of correlation-matching was identified between the bacillary indices obtained in order to assess the degree of interchangeability. Materials and methods. Standardization was attained by a reading of the smears by 2 readers with subsequent, blinded evaluation of inter-observer agreement. Each reader quantified the bacterial load of for each sample (n=325) using the Colombian and the Ridley scales. The degree of interobserver agreement was assessed with weighted kappa coefficient. The level of correlation and agreement between the measurements of the bacillary index was established with coefficient of Lin. Results. The interobserver weighted kappa coefficient was 0.83 for the Colombia scale and 0.85 for the Ridley scale. The Lin coefficient was 0.96 for the correlation-matching of bacillary indexes. Conclusions. Interobserver agreement obtained for both scales was excellent as the correlation-matching bacillary indices determined with both methods. With the cut-off points yielded a good level of agreement, ensuring interchangeability between the scales defining the high or low bacterial load.     

Physicians in health care management: 4. Case Mix Groups and Resource Intensity Weights: physicians and hospital funding.

In the second of two articles on Case Mix Groups (CMGs) and Resource Intensity Weights (RIWs) the authors describe how these measures are used to adjust the funding of hospitals in Ontario. Because CMGs and RIWs are based on medical chart information concerning diagnoses, concurrent illnesses and main procedures the role of physicians in recording this information is important to the outcome for hospital funding. CMGs and RIWs provide the basis for the calculations of the average cost per weighted case for hospitals and for groups of comparable hospitals. The Ontario Ministry of Health originally gave equity adjustment payments to hospitals with low average costs per weighted case to raise their funding toward norms of comparable hospitals. However, it is now proposed that hospitals with high average costs per weighted case be targeted for budget cuts. In the face of greater case-mix-based hospital funding in the future physician recording of information will be ever more critical.     

Comparison of three different ways of measuring distances between residences and high voltage power lines

The aim of this study was to evaluate whether distance data based on calculations by use of digitalized geographical information systems (GIS) and distance data based on measurements on 1:5000 maps agree sufficiently with on site distance measurements to be used as input to magnetic field calculations in epidemiological studies. The analysis were performed by use of weighted kappa (kappa(w)) statistical method described by Bland and Altman for comparison of measures of agreement. Map measurements showed better agreement with on site measurements than GIS calculations did. However, we consider both methods appropriate for use in larger epidemiological studies if the results are interpreted with caution. GIS calculations have the advantage of being both time and cost saving.     

R-body-producing bacteria.

Until 10 years ago, R bodies were known only as diagnostic features by which endosymbionts of paramecia were identified as kappa particles. They were thought to be limited to the cytoplasm of two species in the Paramecium aurelia species complex. Now, R bodies have been found in free-living bacteria and other Paramecium species. The organisms now known to form R bodies include the cytoplasmic kappa endosymbionts of P. biaurelia and P. tetraurelia, the macronuclear kappa endosymbionts of P. caudatum, Pseudomonas avenae (a free-living plant pathogen), Pseudomonas taeniospiralis (a hydrogen-oxidizing soil microorganism), Rhodospirillum centenum (a photosynthetic bacterium), and a soil bacterium, EPS-5028, which is probably a pseudomonad. R bodies themselves fall into five distinct groups, distinguished by size, the morphology of the R-body ribbons, and the unrolling behavior of wound R bodies. In recent years, the inherent difficulties in studying the organization and assembly of R bodies by the obligate endosymbiont kappa, have been alleviated by cloning and expressing genetic determinants for these R bodies (type 51) in Escherichia coli. Type 51 R-body synthesis requires three low-molecular-mass polypeptides. One of these is modified posttranslationally, giving rise to 12 polypeptide species, which are the major structural subunits of the R body. R bodies are encoded in kappa species by extrachromosomal elements. Type 51 R bodies, produced in Caedibacter taeniospiralis, are encoded by a plasmid, whereas bacteriophage genomes probably control R-body synthesis in other kappa species. However, there is no evidence that either bacteriophages or plasmids are present in P. avenae or P. taeniospiralis. No sequence homology was detected between type 51 R-body-encoding DNA and DNA from any R-body-producing species, except C. varicaedens 1038. The evolutionary relatedness of different types of R bodies remains unknown.     

Evolution of a plastic quantitative trait in an age-structured population in a fluctuating environment

We analyze weak fluctuating selection on a quantitative character in an age-structured population not subject to density regulation. We assume that early in the first year of life before selection, during a critical state of development, environments exert a plastic effect on the phenotype, which remains constant throughout the life of an individual. Age-specific selection on the character affects survival and fecundity, which have intermediate optima subject to temporal environmental fluctuations with directional selection in some age classes as special cases. Weighting individuals by their reproductive value, as suggested by Fisher, we show that the expected response per year in the weighted mean character has the same form as for models with no age structure. Environmental stochasticity generates stochastic fluctuations in the weighted mean character following a first-order autoregressive model with a temporally autocorrelated noise term and stationary variance depending on the amount of phenotypic plasticity. The parameters of the process are simple weighted averages of parameters used to describe age-specific survival and fecundity. The "age-specific selective weights" are related to the stable distribution of reproductive values among age classes. This allows partitioning of the change in the weighted mean character into age-specific components.     

R-body-producing bacteria.

Until 10 years ago, R bodies were known only as diagnostic features by which endosymbionts of paramecia were identified as kappa particles. They were thought to be limited to the cytoplasm of two species in the Paramecium aurelia species complex. Now, R bodies have been found in free-living bacteria and other Paramecium species. The organisms now known to form R bodies include the cytoplasmic kappa endosymbionts of P. biaurelia and P. tetraurelia, the macronuclear kappa endosymbionts of P. caudatum, Pseudomonas avenae (a free-living plant pathogen), Pseudomonas taeniospiralis (a hydrogen-oxidizing soil microorganism), Rhodospirillum centenum (a photosynthetic bacterium), and a soil bacterium, EPS-5028, which is probably a pseudomonad. R bodies themselves fall into five distinct groups, distinguished by size, the morphology of the R-body ribbons, and the unrolling behavior of wound R bodies. In recent years, the inherent difficulties in studying the organization and assembly of R bodies by the obligate endosymbiont kappa, have been alleviated by cloning and expressing genetic determinants for these R bodies (type 51) in Escherichia coli. Type 51 R-body synthesis requires three low-molecular-mass polypeptides. One of these is modified posttranslationally, giving rise to 12 polypeptide species, which are the major structural subunits of the R body. R bodies are encoded in kappa species by extrachromosomal elements. Type 51 R bodies, produced in Caedibacter taeniospiralis, are encoded by a plasmid, whereas bacteriophage genomes probably control R-body synthesis in other kappa species. However, there is no evidence that either bacteriophages or plasmids are present in P. avenae or P. taeniospiralis. No sequence homology was detected between type 51 R-body-encoding DNA and DNA from any R-body-producing species, except C. varicaedens 1038. The evolutionary relatedness of different types of R bodies remains unknown.