A cautionary note on the robustness of latent class models for estimating diagnostic error without a gold standard

Modeling diagnostic error without a gold standard has been an active area of biostatistical research. In a majority of the approaches, model-based estimates of sensitivity, specificity, and prevalence are derived from a latent class model in which the latent variable represents an individual's true unobserved disease status. For simplicity, initial approaches assumed that the diagnostic test results on the same subject were independent given the true disease status (i.e., the conditional independence assumption). More recently, various authors have proposed approaches for modeling the dependence structure between test results given true disease status. This note discusses a potential problem with these approaches. Namely, we show that when the conditional dependence between tests is misspecified, estimators of sensitivity, specificity, and prevalence can be biased. Importantly, we demonstrate that with small numbers of tests, likelihood comparisons and other model diagnostics may not be able to distinguish between models with different dependence structures. We present asymptotic results that show the generality of the problem. Further, data analysis and simulations demonstrate the practical implications of model misspecification. Finally, we present some guidelines about the use of these models for practitioners.     

Nonparametric estimation of ROC curves in the absence of a gold standard

In the evaluation of diagnostic accuracy of tests, a gold standard on the disease status is required. However, in many complex diseases, it is impossible or unethical to obtain such a gold standard. If an imperfect standard is used, the estimated accuracy of the tests would be biased. This type of bias is called imperfect gold standard bias. In this article we develop a nonparametric maximum likelihood method for estimating ROC curves and their areas of ordinal-scale tests in the absence of a gold standard. Our simulation study shows that the proposed estimators for the ROC curve areas have good finite-sample properties in terms of bias and mean squared error. Further simulation studies show that our nonparametric approach is comparable to the binormal parametric method, and is easier to implement. Finally, we illustrate the application of the proposed method in a real clinical study on assessing the accuracy of seven specific pathologists in detecting carcinoma in situ of the uterine cervix.     

Imputation approaches for estimating diagnostic accuracy for multiple tests from partially verified designs

Interest often focuses on estimating sensitivity and specificity of a group of raters or a set of new diagnostic tests in situations in which gold standard evaluation is expensive or invasive. Various authors have proposed semilatent class modeling approaches for estimating diagnostic accuracy in this situation. This article presents imputation approaches for this problem. I show how imputation provides a simpler way of performing diagnostic accuracy and prevalence estimation than the use of semilatent modeling. Furthermore, the imputation approach is more robust to modeling assumptions and, in general, there is only a moderate efficiency loss relative to a correctly specified semilatent class model. I apply imputation to a study designed to estimate the diagnostic accuracy of digital radiography for gastric cancer. The feasibility and robustness of imputation is illustrated with analysis, asymptotic results, and simulations.     

Bayesian approaches to modeling the conditional dependence between multiple diagnostic tests

Many analyses of results from multiple diagnostic tests assume the tests are statistically independent conditional on the true disease status of the subject. This assumption may be violated in practice, especially in situations where none of the tests is a perfectly accurate gold standard. Classical inference for models accounting for the conditional dependence between tests requires that results from at least four different tests be used in order to obtain an identifiable solution, but it is not always feasible to have results from this many tests. We use a Bayesian approach to draw inferences about the disease prevalence and test properties while adjusting for the possibility of conditional dependence between tests, particularly when we have only two tests. We propose both fixed and random effects models. Since with fewer than four tests the problem is nonidentifiable, the posterior distributions are strongly dependent on the prior information about the test properties and the disease prevalence, even with large sample sizes. If the degree of correlation between the tests is known a priori with high precision, then our methods adjust for the dependence between the tests. Otherwise, our methods provide adjusted inferences that incorporate all of the uncertainty inherent in the problem, typically resulting in wider interval estimates. We illustrate our methods using data from a study on the prevalence of Strongyloides infection among Cambodian refugees to Canada.     

Sensitivity of diagnostic tests for human soil-transmitted helminth infections: a meta-analysis in the absence of a true gold standard

Reliable, sensitive and practical diagnostic tests are an essential tool in disease control programmes for mapping, impact evaluation and surveillance. To provide a robust global assessment of the relative performance of available diagnostic tools for the detection of soil-transmitted helminths, we conducted a meta-analysis comparing the sensitivities and the quantitative performance of the most commonly used copro-microscopic diagnostic methods for soil-transmitted helminths, namely Kato-Katz, direct microscopy, formol-ether concentration, McMaster, FLOTAC and Mini-FLOTAC. In the absence of a perfect reference standard, we employed a Bayesian latent class analysis to estimate the true, unobserved sensitivity of compared diagnostic tests for each of the soil-transmitted helminth species Ascaris lumbricoides, Trichuris trichiura and the hookworms. To investigate the influence of varying transmission settings we subsequently stratified the analysis by intensity of infection. Overall, sensitivity estimates varied between the different methods, ranging from 42.8% for direct microscopy to 92.7% for FLOTAC. The widely used double slide Kato-Katz method had a sensitivity of 74–95% for the three soil-transmitted helminth species at high infection intensity, however sensitivity dropped to 53–80% in low intensity settings, being lowest for hookworm and A. lumbricoides. The highest sensitivity, overall and in both intensity groups, was observed for the FLOTAC method, whereas the sensitivity of the Mini-FLOTAC method was comparable with the Kato-Katz method. FLOTAC average egg count estimates were significantly lower compared with Kato-Katz, while the compared McMaster counts varied. In conclusion, we demonstrate that the Kato-Katz and Mini-FLOTAC methods had comparable sensitivities. We further show that test sensitivity of the Kato-Katz method is reduced in low transmission settings.