| Abstract: | The quality of ordered categorical recordings is determined from repeated measurements on the same subject in order to assess the level of agreement between raters, scales or occasions. The presented rating-invariant method for ordered categorical data provides means of analysing the quality of single-item rating scales, irrespective of the number of possible response values and the marginal distributions. Marginal heterogeneity implies systematic disagreement, so-called bias. An augmented ranking approach is the basis for the separation of inter-rater disagreement into systematic and random components. Correlation between pairs of augmented rank values provides a measure of agreement to the best common ordering of paired classifications, given inter-rater bias. The essential differences in interpretation and applicability of the proposed coefficient of agreement and the Spearman rank-order correlation for ordered categorical data are discussed. |
