Evaluating the effects of rater and subject factors on measures of association

Large‐scale agreement studies are becoming increasingly common in medical settings to gain better insight into discrepancies often observed between experts' classifications. Ordered categorical scales are routinely used to classify subjects' disease and health conditions. Summary measures such as Cohen's weighted kappa are popular approaches for reporting levels of association for pairs of raters' ordinal classifications. However, in large‐scale studies with many raters, assessing levels of association can be challenging due to dependencies between many raters each grading the same sample of subjects' results and the ordinal nature of the ratings. Further complexities arise when the focus of a study is to examine the impact of rater and subject characteristics on levels of association. In this paper, we describe a flexible approach based upon the class of generalized linear mixed models to assess the influence of rater and subject factors on association between many raters' ordinal classifications. We propose novel model‐based measures for large‐scale studies to provide simple summaries of association similar to Cohen's weighted kappa while avoiding prevalence and marginal distribution issues that Cohen's weighted kappa is susceptible to. The proposed summary measures can be used to compare association between subgroups of subjects or raters. We demonstrate the use of hypothesis tests to formally determine if rater and subject factors have a significant influence on association, and describe approaches for evaluating the goodness‐of‐fit of the proposed model. The performance of the proposed approach is explored through extensive simulation studies and is applied to a recent large‐scale cancer breast cancer screening study.     

Powerful Exact Unconditional Tests for Agreement between Two Raters with Binary Endpoints

Asymptotic and exact conditional approaches have often been used for testing agreement between two raters with binary outcomes. The exact conditional approach is guaranteed to respect the test size as compared to the traditionally used asymptotic approach based on the standardized Cohen''s kappa coefficient. An alternative to the conditional approach is an unconditional strategy which relaxes the restriction of fixed marginal totals as in the conditional approach. Three exact unconditional hypothesis testing procedures are considered in this article: an approach based on maximization, an approach based on the conditional p-value and maximization, and an approach based on estimation and maximization. We compared these testing procedures based on the commonly used Cohen''s kappa with regards to test size and power. We recommend the following two exact approaches for use in practice due to power advantages: the approach based on conditional p-value and maximization and the approach based on estimation and maximization.     

Poor-to-moderate agreement between self and proxy interviews of mobile phone use

In epidemiological studies, cases cannot always be interviewed due to them being too ill or already deceased. Under these circumstances, proxy interviews are often conducted; however, the veridicality of information about mobile phone use gained by proxy interviews has been doubted. The issue is undecided due to the lack of empirical data. We conducted a study of 119 heterosexual couples. Both partners answered two questionnaires about mobile phone use, one about their own use and one about their partner's use. Overall agreement assessed using Cohen's kappa, Passing and Bablok regression, and concordance coefficients between self and proxy data was poor to moderate (e.g., concordance coefficients of 0.55 for duration of use). The only item with good agreement was whether or not a prepaid phone was used (Cohen's kappa 0.78 and 0.63 for male and female estimates, respectively), and to a lesser degree, the onset of mobile phone use (concordance coefficients of 0.66 and 0.61). Poorest agreement was obtained for the side of the head the mobile phone was held during calls (kappa coefficients of 0.20 and 0.24 for female and male estimates, respectively). We conclude that the assessment of mobile phone use by proxy data cannot be relied on except for information about onset of mobile phone use, use of prepaid or contract phones, and, to a lesser degree, duration of daily use. Agreement concerning the important information about side of the head the mobile phone is held during calls was poorest and only slightly better than chance. Bioelectromagnetics 33:561–567, 2012. © 2012 Wiley Periodicals, Inc.     

Diagnostic value of microtympanometry in primary care.

OBJECTIVE--To determine the reliability, validity, and feasibility of a new hand held microtympanometer. DESIGN--Comparison of microtympanometry by two independent observations of a general practitioner and a nurse, and against a validated reference instrument. SETTING--Primary care health centre of a school for the deaf in the United States. SUBJECTS--111 schoolchildren receiving a regular check up. MAIN OUTCOME MEASURES--Tympanometry with the Grason Stadler 28, classified with a slightly modified Jerger''s classification. RESULTS--Interobserver reliability was 0.95 (Cohen''s kappa). Results of microtympanometry were highly comparable with results of the reference instrument (likelihood ratio of positive results, 161.2). CONCLUSIONS--The microtympanometer could be used in general practice: it is hand held, child friendly, easy to handle, and accurate.     

How to obtain NNT from Cohen's d: comparison of two methods

Background

In the literature we find many indices of size of treatment effect (effect size: ES). The preferred index of treatment effect in evidence-based medicine is the number needed to treat (NNT), while the most common one in the medical literature is Cohen''s d when the outcome is continuous. There is confusion about how to convert Cohen''s d into NNT.

Methods

We conducted meta-analyses of individual patient data from 10 randomized controlled trials of second generation antipsychotics for schizophrenia (n = 4278) to produce Cohen''s d and NNTs for various definitions of response, using cutoffs of 10% through 90% reduction on the symptom severity scale. These actual NNTs were compared with NNTs calculated from Cohen''s d according to two proposed methods in the literature (Kraemer, et al., Biological Psychiatry, 2006; Furukawa, Lancet, 1999).

Results

NNTs from Kraemer''s method overlapped with the actual NNTs in 56%, while those based on Furukawa''s method fell within the observed ranges of NNTs in 97% of the examined instances. For various definitions of response corresponding with 10% through 70% symptom reduction where we observed a non-small number of responders, the degree of agreement for the former method was at a chance level (ANOVA ICC of 0.12, p = 0.22) but that for the latter method was ANOVA ICC of 0.86 (95%CI: 0.55 to 0.95, p<0.01).

Conclusions

Furukawa''s method allows more accurate prediction of NNTs from Cohen''s d. Kraemer''s method gives a wrong impression that NNT is constant for a given d even when the event rate differs.     

Use of the Maximum Likelihood Method in the Analysis of Phenotypic Stability

The stability variance is an important estimator of phenotypic stability of genotypes. It may be estimated by method of moments and by maximum likelihood. We demonstrate by Monte Carlo simulation that, given a sufficient number of environments, maximum likelihood estimates (MLE's) are slightly better if ranking of genotypes is the experimenter's major aim. A likelihood ratio test is available for different hypotheses.     

Introducing the vectorial Kappa: An index to quantify congruence between vectorial mosaics

This paper introduces the vectorial Kappa (κ_v) that one can utilize to assess congruence between two vectorial mosaics. The vectorial Kappa extends for vectorial mosaics the approach of the so-called Cohen's Kappa index, commonly used to compare raster mosaics. By comparing both approaches, we aim to demonstrate how efficient and convenient a vector-based congruence may be when working on vectorial mosaics.     

Detecting subtle changes in visuospatial executive function and learning in the amnestic variant of mild cognitive impairment

Background and Purpose

Amnestic mild cognitive impairment (aMCI) is a putative prodromal stage of Alzheimer''s disease (AD) characterized by deficits in episodic verbal memory. Our goal in the present study was to determine whether executive dysfunction may also be detectable in individuals diagnosed with aMCI.

Methods

This study used a hidden maze learning test to characterize component processes of visuospatial executive function and learning in a sample of 62 individuals with aMCI compared with 94 healthy controls.

Results

Relative to controls, individuals with aMCI made more exploratory/learning errors (Cohen''s d = .41). Comparison of learning curves revealed that the slope between the first two of five learning trials was four times as steep for controls than for individuals with aMCI (Cohen''s d = .64). Individuals with aMCI also made a significantly greater number of rule-break/error monitoring errors across learning trials (Cohen''s d = .21).

Conclusions

These results suggest that performance on a task of complex visuospatial executive function is compromised in individuals with aMCI, and likely explained by reductions in initial strategy formulation during early visual learning and “on-line” maintenance of task rules.     

Concordance between Patient Self-Reports and Claims Data on Clinical Diagnoses,Medication Use,and Health System Utilization in Taiwan

Purpose

The aim of this study was to evaluate the concordance between claims records in the National Health Insurance Research Database and patient self-reports on clinical diagnoses, medication use, and health system utilization.

Methods

In this study, we used the data of 15,574 participants collected from the 2005 Taiwan National Health Interview Survey. We assessed positive agreement, negative agreement, and Cohen''s kappa statistics to examine the concordance between claims records and patient self-reports.

Results

Kappa values were 0.43, 0.64, and 0.61 for clinical diagnoses, medication use, and health system utilization, respectively. Using a strict algorithm to identify the clinical diagnoses recorded in claims records could improve the negative agreement; however, the effect on positive agreement and kappa was diverse across various conditions.

Conclusion

We found that the overall concordance between claims records in the National Health Insurance Research Database and patient self-reports in the Taiwan National Health Interview Survey was moderate for clinical diagnosis and substantial for both medication use and health system utilization.     

Do U.S. county data disprove linear no‐threshold predictions of lung cancer risk for residential radon?‐A preliminary assessment of biological plausibility

Lung‐cancer mortality (LCM) is elevated in underground miners who chronically inhaled the mutagenic, cytotoxic α‐decay products of radon gas. Epidemiologie studies of LCM rates vs. residential‐radon concentration levels are generally considered inconclusive. However, Cohen (Health Physics 68, 157–174, 1995) has hypothesized that data on LCM vs. residential radon concentrations at the U.S. county level are clearly inconsistent with a linear no‐threshold (LN) dose‐response model, and rather are consistent with threshold or hormesis model. Cohen's hypothesis has been criticized as “ecological fallacy,”; particularly because LN (but not threshold or hormesis) models are generally considered biologically plausible for agents like α radiation that damage DNA in linear proportion to dose. To assess the biological plausibility of Cohen's hypothesis, a preliminary study was made of whether a biologically realistic, cytodynamic 2‐stage (CD2) cancer model can provide a good, joint fit to Cohen's set of U.S. county data as well as to underground‐miner data. The CD2 model used adapts a widely applied, mechanistic, 2‐stage stochastic model of carcinogenesis to realistically account for interrelated cell killing and mutation (both assumed to have a LN dose‐response), cell turnover, and incomplete exposure of stem cells. A CD2 fit was obtained to combined summary data on LCM vs. radon‐exposure in white males in 1, 601 U.S. counties (from Cohen) and in white male Colorado Plateau (CP) uranium miners (from the National Research Council's “BEIRIV”; report). The CD2 fit is shown to: (i) be consistent with the combined data; (ii) have parameter values all consistent with biological data; and (iii) predict inverse dose‐rate‐effects data for CP and other radon‐exposed miners, despite the fact that optimization had not involved any of these dose‐rate data. The latter data were not predicted by a simplified CD2 model in which all stem cells were presumed to be exposed. It is concluded that this study provides preliminary evidence that Cohen's hypothesis is biologically plausible.     

Phlebotominae of epidemiological importance in cutaneous leishmaniasis in northwestern Argentina: risk maps and ecological niche models

In Argentina, 58.2% out of the 8126 Cutaneous Leishmaniasis (CL) incident cases accumulated from 1954 to 2006 were reported in the provinces of Salta and Jujuy. The aim of this study was to develop an exploratory risk map and a potential distribution map of the vector, in order to offer recommendations for CL prevention. A total of 12 079 Phlebotominae (Diptera: Psychodidae) belonging to the species Lutzomyia neivai (Pinto), Lu. migonei (França), Lu. cortelezzii (Brèthes), Lu. shannoni (Dyar), Lu. quinquefer (Dyar) and Brumptomyia spp. (França & Parrot) were captured. Potential distribution models were created for two species, Lu. neivai (incriminated vector of Leishmania braziliensis) and Lu. migonei, associated with domestic animals in Argentina and that in turn could be involved as a link between zoonotic transmission cycles and anthropozoonotic. The Maximum Entropy Modeling System (MaxEnt) was used. The Jackknife test was performed, and the ‘rainfall of the driest month’ was the variable that best generalized the models. Accuracy was evaluated by the area under the curve (AUC) and validated by the Cohen's kappa index. This approximation provides a new analytical resource of high potential for the prevention of the disease, in order to allocate resources properly and to develop the most suitable strategies for action.     

Effective group training for patients with unexplained physical symptoms: a randomized controlled trial with a non-randomized one-year follow-up

Background

Although cognitive-behavioral therapy for Unexplained Physical Symptoms (UPS) is effective in secondary care, studies done in primary care produced implementation problems and conflicting results. We evaluated the effectiveness of a cognitive-behavioral group training tailored to primary care patients and provided by a secondary community mental-health service reaching out into primary care.

Methodology/Principal Findings

The effectiveness of this training was explored in a randomized controlled trial. In this trial, 162 patients with UPS classified as undifferentiated somatoform disorder or as chronic pain disorder were randomized either to the training or a waiting list. Both lasted 13 weeks. The preservation of the training''s effect was analyzed in non-randomized follow-ups, for which the waiting group started the training after the waiting period. All patients attended the training were followed-up after three months and again after one year. The primary outcomes were the physical and the mental summary scales of the SF-36. Secondary outcomes were the other SF-36-scales and the SCL-90-R. The courses of the training''s effects in the randomized controlled trial and the follow-ups were analyzed with linear mixed modeling. In the randomized controlled trial, the training had a significantly positive effect on the quality of life in the physical domain (Cohen''s d = 0.38;p = .002), but this overall effect was not found in the mental domain. Regarding the secondary outcomes, the training resulted in reporting an improved physical (Cohen''s d = 0.43;p = 0.01), emotional (Cohen''s d = 0.44;p = .0.01), and social (Cohen''s d = 0.36;p = 0.01) functioning, less pain and better functioning despite pain (Cohen''s d = 0.51;p = <0.001), less physical symptoms (Cohen''s d = −.23;p = 0.05) and less sleep difficulties (Cohen''s d = −0.25;p = 0.04) than time in the waiting group. During the non-randomized follow-ups, there were no relapses.

Conclusions/Significance

The cognitive-behavioral group training tailored for UPS in primary care and provided by an outreaching secondary mental-health service appears to be effective and to broaden the accessibility of treatment for UPS.

Trial Registration

TrialRegister.nl NTR1609 <rctview.asp?TC = 1609>     

Wild mink (Neovison vison) as sentinels in environmental monitoring

Superficial inguinal lymph nodes from 72 wild boars examined in a previous immunohistochemical (IHC) study on porcine circovirus type 2 (PCV2) were selected for a PCV2 polymerase chain reaction (PCR) analysis. Four of these lymph nodes were PCV2-IHC strongly positive with PMWS histological lesions (outcome 1), 6 weak to mild PCV2-IHC positive without PMWS histological lesions (outcome 2) and 62 PCV2-IHC negative. Considering IHC the gold standard for diagnosis, the aims of the study were to evaluate the suitability of the PCV2-DNA extraction from formalin-fixed and paraffin-embedded (FFPE) tissue and the sensitivity and specificity of PCR under two IHC interpretations criteria: (A) the sample was considered positive if the result was outcome 1; (B) the sample was considered positive if the result was outcome 1 or 2. Under (A) criteria, sensitivity and specificity of PCR were 100% and 89.7%, respectively; the Cohen's Kappa coefficient was 0.49. Under (B) criteria, sensitivity and specificity of PCR were 80.0% and 95.2%, respectively; the Cohen's Kappa coefficient was 0.72. The high Cohen's Kappa coefficient under the (B) interpretative criteria indicates good agreement between the two methods. In conclusion, 1) DNA extracted from FFPE specimens of wild boar is suitable for PCR and further represents a screening test for PCV2/PCVD (PCV2 Diseases) investigations in wild boar as well; 2) routine histological sampling can also be useful for PCV2 virological studies in wild boar.     

A Class of Ratio Cum Product-Type Estimator in Double Sampling

A class of ratio cum product-type estimator is proposed in case of double sampling in the present paper. Its bias and variance to the first order of approximation are obtained. For an appropriate weight ‘a’ and a good range of α-values, it is found that the proposed estimator is more efficient than the set of estimator viz., simple mean estimator, usual ratio and product estimators, SRIVASTAVA 's estimator (1967), CHAKARBARTY 's estimator and product-type estimator, which are in fact the particular cases of it. The proposed estimator is as efficient as linear regression estimator in double sampling at optimum value of α.     

A model for radiofrequency electromagnetic field predictions at outdoor and indoor locations in the context of epidemiological research

We present a geospatial model to predict the radiofrequency electromagnetic field from fixed site transmitters for use in epidemiological exposure assessment. The proposed model extends an existing model toward the prediction of indoor exposure, that is, at the homes of potential study participants. The model is based on accurate operation parameters of all stationary transmitters of mobile communication base stations, and radio broadcast and television transmitters for an extended urban and suburban region in the Basel area (Switzerland). The model was evaluated by calculating Spearman rank correlations and weighted Cohen's kappa (κ) statistics between the model predictions and measurements obtained at street level, in the homes of volunteers, and in front of the windows of these homes. The correlation coefficients of the numerical predictions with street level measurements were 0.64, with indoor measurements 0.66, and with window measurements 0.67. The kappa coefficients were 0.48 (95%‐confidence interval: 0.35–0.61) for street level measurements, 0.44 (95%‐CI: 0.32–0.57) for indoor measurements, and 0.53 (95%‐CI: 0.42–0.65) for window measurements. Although the modeling of shielding effects by walls and roofs requires considerable simplifications of a complex environment, we found a comparable accuracy of the model for indoor and outdoor points. Bioelectromagnetics 31:226–236, 2010. © 2009 Wiley‐Liss, Inc.     

Effect sizes in experimental pain produced by gender, genetic variants and sensitization procedures

Background

Various effects on pain have been reported with respect to their statistical significance, but a standardized measure of effect size has been rarely added. Such a measure would ease comparison of the magnitude of the effects across studies, for example the effect of gender on heat pain with the effect of a genetic variant on pressure pain.

Methodology/Principal Findings

Effect sizes on pain thresholds to stimuli consisting of heat, cold, blunt pressure, punctuate pressure and electrical current, administered to 125 subjects, were analyzed for 29 common variants in eight human genes reportedly modulating pain, gender and sensitization procedures using capsaicin or menthol. The genotype explained 0–5.9% of the total interindividual variance in pain thresholds to various stimuli and produced mainly small effects (Cohen''s d 0–1.8). The largest effect had the TRPA1 rs13255063T/rs11988795G haplotype explaining >5% of the variance in electrical pain thresholds and conferring lower pain sensitivity to homozygous carriers. Gender produced larger effect sizes than most variant alleles (1–14.8% explained variance, Cohen''s d 0.2–0.8), with higher pain sensitivity in women than in men. Sensitization by capsaicin or menthol explained up to 63% of the total variance (4.7–62.8%) and produced largest effects according to Cohen''s d (0.4–2.6), especially heat sensitization by capsaicin (Cohen''s d = 2.6).

Conclusions

Sensitization, gender and genetic variants produce effects on pain in the mentioned order of effect sizes. The present report may provide a basis for comparative discussions of factors influencing pain.     

A Class of Ratio-Cum-Product-Type Estimator

The problem of estimating the population mean using an auxiliary information has been dealt with in literature quite extensively. Ratio, product, linear regression and ratio-type estimators are well known. A class of ratio-cum-product-type estimator is proposed in this paper. Its bias and variance to the first order of approximation are obtained. For an appropriate weight ‘a’ and good range of α-values, it is found that the proposed estimator is superior than a set of estimators (i.e., sample mean, usual ratio and product estimators, SRIVASTAVA's (1967) estimator, CHAKRABARTY's (1979) estimator and a product-type estimator) which are, in fact, the particular cases of it. At optimum value of α, the proposed estimator is as efficient as linear regression estimator.     

Sleep stages classification based on heart rate variability and random forest

An alternative technique for sleep stages classification based on heart rate variability (HRV) was presented in this paper. The simple subject specific scheme and a more practical subject independent scheme were designed to classify wake, rapid eye movement (REM) sleep and non-REM (NREM) sleep. 41 HRV features extracted from RR sequence of 45 healthy subjects were trained and tested through random forest (RF) method. Among the features, 25 were newly proposed or applied to sleep study for the first time. For the subject independent classifier, all features were normalized with our developed fractile values based method. Besides, the importance of each feature for sleep staging was also assessed by RF and the appropriate number of features was explored. For the subject specific classifier, a mean accuracy of 88.67% with Cohen's kappa statistic κ of 0.7393 was achieved. While the accuracy and κ dropped to 72.58% and 0.4627, respectively when the subject independent classifier was considered. Some new proposed HRV features even performed more effectively than the conventional ones. The proposed method could be used as an alternative or aiding technique for rough and convenient sleep stages classification.