Comparisons of predictive values of binary medical diagnostic tests for paired designs

Positive and negative predictive values of a diagnostic test are key clinically relevant measures of test accuracy. Surprisingly, statistical methods for comparing tests with regard to these parameters have not been available for the most common study design in which each test is applied to each study individual. In this paper, we propose a statistic for comparing the predictive values of two diagnostic tests using this paired study design. The proposed statistic is a score statistic derived from a marginal regression model and bears some relation to McNemar's statistic. As McNemar's statistic can be used to compare sensitivities and specificities of diagnostic tests, parameters that condition on disease status, our statistic can be considered as an analog of McNemar's test for the problem of comparing predictive values, parameters that condition on test outcome. We report on the results of a simulation study designed to examine the properties of this test under a variety of conditions. The method is illustrated with data from a study of methods for diagnosis of coronary artery disease.     

The effect of marker choice on estimated levels of introgression across an avian (Pipridae: Manacus) hybrid zone

Hybrid zones are often characterized by narrow, coincident clines for diverse traits, suggesting that little introgression occurs across them. However, this pattern may result from a bias in focussing on traits that are diagnostic of parental populations. Such choice of highly differentiated traits may cause us to overlook differential introgression in nondiagnostic traits and to distort our perception of hybrid zones. We tested this hypothesis in an avian hybrid zone by comparing cline structure in two sets of molecular markers: isozyme and restriction fragment length polymorphism markers chosen for differentiation between parental forms, and microsatellite markers chosen for polymorphism. Two cline‐fitting methods showed that cline centre positions of microsatellite alleles were more variable than those of isozyme and restriction fragment length polymorphism markers, and several were significantly shifted from those of the diagnostic markers. Cline widths of microsatellite alleles were also variable and two‐ to eightfold wider than those of the diagnostic markers. These patterns are consistent with the idea that markers chosen for differentiation are more likely to be under purifying selection, and studies focussed on these markers will underestimate overall introgression across hybrid zones. Our results suggest that neutral and positively selected alleles may introgress freely across many hybrid zones without altering perceived boundaries between hybridizing forms.     

Efficient strategies for genomic searching using the affected-pedigree-member method of linkage analysis.

The affected-pedigree-member (APM) method of linkage analysis is a nonparametric statistic that tests for nonrandom cosegregation of a disease and marker loci. The APM statistic is based on the observation that if a marker locus is near a disease-susceptibility locus, then affected individuals within a family should be more similar at the marker locus than is expected by chance. The APM statistic measures marker similarity in terms of identity by state (IBS) of marker alleles; that is, two alleles are IBS if they are the same, regardless of their ancestral origin. Since the APM statistic measures increased marker similarity, it makes no assumptions concerning how the disease is inherited; this can be an advantage when dealing with complex diseases for which the mode of inheritance is difficult to determine. We investigate here the power of the APM statistic to detect linkage in the context of a genomewide search. In such a search, the APM statistic is evaluated at a grid of markers. Then regions with high APM statistics are investigated more thoroughly by typing more markers in the region. Using simulated data, we investigate various search strategies and recommend an optimal search strategy that maximizes the power to detect linkage while minimizing the false-positive rate and number of markers. We determine an optimal series of three increasing cut-points and an independent criterion for significance.     

Comparison of ROC umbrella volumes with an application to the assessment of lung cancer diagnostic markers

Receiver operating characteristic (ROC) analysis is widely used to assess the ability of diagnostic markers to correctly classify into one of two disease classes. ROC surfaces and umbrella surfaces generalize the utility of ROC analysis when there are three disease classes. Identification of lung cancer diagnostic markers is an active area of research since prognosis for those diagnosed with lung cancer is so poor and there is not an accurate method for early detection of lung cancer. A study conducted for the assessment of DNA methylation markers motivated the comparison of ROC umbrella surfaces which is developed in this article using U-statistics and bootstrap methodology.     

The diagnostic performance of current tumour markers in surveillance for recurrent testicular cancer: A diagnostic test accuracy systematic review

In this diagnostic test accuracy systematic review we summarise the evidence on the diagnostic accuracy of blood α-fetoprotein (AFP), human chorionic gonadotropin (HCG) and lactate dehydrogenase (LDH) in surveillance for testicular cancer recurrence in adults. We searched four electronic databases for studies that reported the diagnostic accuracy of HCG, AFP, and/or LDH in sufficient detail for sensitivity and specificity to be calculated by extracting a 2 × 2 table comparing biomarker positivity with testicular cancer recurrence. Screening, data extraction and QUADAS-2 quality assessment were completed by two independent reviewers. From 2406 studies, nine met our inclusion criteria. Eight reported data at the per-patient level. Sample sizes were small (range 5 to 449 patients) and clinical heterogeneity precluded meta-analysis. In most studies the specificity for recurrence with AFP and HCG was high (90–100%) but sensitivity was often relatively low, suggesting that many recurrences would not be detected by tumour markers alone. The diagnostic performance of LDH appears poorer. Studies were methodologically weak, with probable selection, incorporation and partial verification bias, and many studies were excluded for not reporting on recurrence-free patients. Limitations including small sample sizes, high heterogeneity, and inconsistent and incomplete reporting mean these results must be interpreted with caution. Despite inclusion of biomarkers in international surveillance guidance, there remains a lack of high quality evidence about their accuracy, optimal thresholds, and the most effective surveillance strategy in relation to contemporary investigative modalities. Higher quality research using data from modern-day follow-up cohorts is necessary to identify opportunities to reduce unnecessary testing.     

Simultaneous Non‐inferiority Test of Sensitivity and Specificity for Two Diagnostic Procedures in the Presence of a Gold Standard

Sensitivity and specificity have traditionally been used to assess the performance of a diagnostic procedure. Diagnostic procedures with both high sensitivity and high specificity are desirable, but these procedures are frequently too expensive, hazardous, and/or difficult to operate. A less sophisticated procedure may be preferred, if the loss of the sensitivity or specificity is determined to be clinically acceptable. This paper addresses the problem of simultaneous testing of sensitivity and specificity for an alternative test procedure with a reference test procedure when a gold standard is present. The hypothesis is formulated as a compound hypothesis of two non‐inferiority (one‐sided equivalence) tests. We present an asymptotic test statistic based on the restricted maximum likelihood estimate in the framework of comparing two correlated proportions under the prospective and retrospective sampling designs. The sample size and power of an asymptotic test statistic are derived. The actual type I error and power are calculated by enumerating the exact probabilities in the rejection region. For applications that require high sensitivity as well as high specificity, a large number of positive subjects and a large number of negative subjects are needed. We also propose a weighted sum statistic as an alternative test by comparing a combined measure of sensitivity and specificity of the two procedures. The sample size determination is independent of the sampling plan for the two tests.     

A new statistic for detecting genetic differentiation

A new statistic for detecting genetic differentiation of subpopulations is described. The statistic can be calculated when genetic data are collected on individuals sampled from two or more localities. It is assumed that haplotypic data are obtained, either in the form of DNA sequences or data on many tightly linked markers. Using a symmetric island model, and assuming an infinite-sites model of mutation, it is found that the new statistic is as powerful or more powerful than previously proposed statistics for a wide range of parameter values.     

Quantitative proteomic analysis to discover potential diagnostic markers and therapeutic targets in human renal cell carcinoma

Renal cell carcinoma (RCC) is relatively resistant to chemotherapy and radiotherapy. Recent advances in drug development are providing novel agents for the treatment of RCC, but the effects are still minimal. In addition, there is an urgent need to identify diagnostic markers for RCC. In this report, to discover potential diagnostic markers and therapeutic targets, we subjected RCC samples to a quantitative proteomic analysis utilizing 2-nitrobenzenesulfenyl (NBS) reagent. Proteins were extracted from RCC and adjacent normal tissue, obtained surgically from patients, and labeled with NBS reagent containing six (12)C or (13)C. This was followed by trypsin digestion and the enrichment of labeled peptides. Samples were then subjected to analysis by MALDI-TOF MS. NBS-labeled peptides with a 6 Da difference were identified by MS/MS. Thirty-four proteins were upregulated in more than 60% of the patients of which some were previously known, and some were novel. The identity of a few proteins was confirmed by Western blotting and quantitative real time RT-PCR. The results suggest that NBS-based quantitative proteomic analysis is useful for discovering diagnostic markers and therapeutic targets for RCC.     

The Skill Plot: a graphical technique for evaluating continuous diagnostic tests

Summary .   We introduce the Skill Plot, a method that it is directly relevant to a decision maker who must use a diagnostic test. In contrast to ROC curves, the skill curve allows easy graphical inspection of the optimal cutoff or decision rule for a diagnostic test. The skill curve and test also determine whether diagnoses based on this cutoff improve upon a naive forecast (of always present or of always absent). The skill measure makes it easy to directly compare the predictive utility of two different classifiers in an analogy to the area under the curve statistic related to ROC analysis. Finally, this article shows that the skill-based cutoff inferred from the plot is equivalent to the cutoff indicated by optimizing the posterior odds in accordance with Bayesian decision theory. A method for constructing a confidence interval for this optimal point is presented and briefly discussed.     

Comparing the likelihood ratios of two binary diagnostic tests in the presence of partial verification

The comparison of the efficiency of two binary diagnostic tests requires one to know the disease status for all patients in the sample, by applying a gold standard. In two-phase studies the gold standard is not applied to all patients in a sample, and the problem of partial verification of the disease arises. At present, one of the approaches most used for comparing two binary diagnostic tests are the likelihood ratios. In this study, the maximum likelihood estimators of likelihood ratios are obtained. The tests of hypothesis to compare the likelihood ratios of two binary diagnostic tests when both are applied to the same random sample in the presence of verification bias are deduced, and simulation experiments are performed in order to investigate the asymptotic behaviour of the tests of hypothesis. The results obtained have been applied to the study of Alzheimer's disease.     

Combining diagnostic test results to increase accuracy

When multiple diagnostic tests are performed on an individual or multiple disease markers are available it may be possible to combine the information to diagnose disease. We consider how to choose linear combinations of markers in order to optimize diagnostic accuracy. The accuracy index to be maximized is the area or partial area under the receiver operating characteristic (ROC) curve. We propose a distribution-free rank-based approach for optimizing the area under the ROC curve and compare it with logistic regression and with classic linear discriminant analysis (LDA). It has been shown that the latter method optimizes the area under the ROC curve when test results have a multivariate normal distribution for diseased and non-diseased populations. Simulation studies suggest that the proposed non-parametric method is efficient when data are multivariate normal.The distribution-free method is generalized to a smooth distribution-free approach to: (i) accommodate some reasonable smoothness assumptions; (ii) incorporate covariate effects; and (iii) yield optimized partial areas under the ROC curve. This latter feature is particularly important since it allows one to focus on a region of the ROC curve which is of most relevance to clinical practice. Neither logistic regression nor LDA necessarily maximize partial areas. The approaches are illustrated on two cancer datasets, one involving serum antigen markers for pancreatic cancer and the other involving longitudinal prostate specific antigen data.     

The diagnostic and molecular characteristics of malignant mesothelioma and ovarian/peritoneal serous carcinoma

B. Davidson The diagnostic and molecular characteristics of malignant mesothelioma and ovarian/peritoneal serous carcinoma Malignant mesothelioma and ovarian/peritoneal serous carcinoma are two of the most common tumours affecting the serosal cavities. Unlike other malignant tumours diagnosed at this anatomical site, such as lung and breast carcinoma, malignant mesothelioma and serous carcinoma share a common histogenesis, may be difficult to differentiate morphologically, and co‐express many of the diagnostic markers used by cytopathologists in effusion diagnosis. Selected markers have nevertheless shown sufficient sensitivity and specificity to differentiate serous carcinoma from malignant mesothelioma effectively. Recently, our group applied high throughput technology to the identification of new markers that may aid in differentiating these two cancer types and validated several of these markers in follow‐up studies. This review will present current data regarding the diagnostic and biological aspects of malignant mesothelioma and ovarian/peritoneal serous carcinoma.     

Juxtaposed microsatellite systems as diagnostic markers for admixture: an empirical evaluation with brown trout (Salmo trutta) as model organism

A juxtaposed microsatellite system (JMS) is composed of two microsatellite repeat arrays separated by a sequence of less than 200 bp and more than 20 bp. This paper presents the first empirical evaluation of JMSs for the study of genetic admixture induced by man, with brown trout (Salmo trutta) as model organism. Two distinct admixture situations were studied: native populations from streams of the Atlantic basin and of the Mediterranean basin, respectively, all stocked with domestic strains originating from the Atlantic basin. For these two situations, we first evaluated by simulation the ability of JMSs to differentiate between alien alleles and naturally shared homoplasious or ancestral alleles, and thus to behave as diagnostic markers for admixture. Simulations indicated that JMSs are expected to be reliable diagnostic markers in most divergent (i.e. Mediterranean) populations and nonreliable diagnostic markers in most closely related (i.e. Atlantic) populations. Three JMSs were genotyped in domestic strains as well as in nonstocked and stocked populations of brown trout sampled in different rivers of the Mediterranean and Atlantic basins. The observed distributions of JMS haplotypes were consistent with simulation predictions confirming that JMSs were reliable diagnostic markers only over a given proportion of the species range, i.e. in substantially divergent populations. JMSs also reinforced the diagnostic character of three microsatellite sites for the studied Mediterranean populations. This last result is consistent with our simulation results which showed that, although much less frequently than at JMSs, diagnostic markers are likely to be found at single site microsatellites provided that the native Mediterranean population has a sufficiently small effective population size. For each population of the Mediterranean basin admixture coefficients did not differ significantly across JMSs and mean admixture coefficients sometimes differ among populations. The interpretation of the origin of JMS haplotypes based on the allele length variants was supported by nucleotide sequence analysis.     

BAC-derived diagnostic markers for sex determination in asparagus

A HindIII BAC (bacterial artificial chromosome) library of asparagus (Asparagus officinalis L.) was established from a single male plant homozygous for the male flowering gene (MM). The library represents approximately 5.5 haploid genome equivalents with an average insert size of 82 kb. A subset of the library (2.6 haploid genome equivalents) was arranged into DNA pools. Using nine sex-linked amplified fragment length polymorphism (AFLP) and two sequence-tagged site (STS) markers, 13 different BAC clones were identified from this part of the library. The BACs were arranged into a first-generation physical map around the sex locus. Four PCR-derived markers were developed from the BAC ends, one of which could be scored in a co-dominant way. Using a mapping population of 802 plants we mapped the BAC-derived markers to the same position close to the M gene as the corresponding AFLP and STS markers. The markers are useful for further chromosome walking studies and as diagnostic markers for selecting male plants homozygous for the M gene.Communicated by G. Wenzel     

How industry is approaching the search for new diagnostic markers and biomarkers

In the diagnostic and the pharmaceutical industry there is a constant need for new diagnostic markers and biomarkers with improved sensitivity and specificity. During the last 5 years, only a few novel diagnostic markers have been introduced into the market. Proteomics technologies are now offering unique chances to identify new candidate markers. Before a marker can be introduced into the market, three successive developmental phases have to be completed: the discovery phase, in which a variety of proteomics technologies are applied to identify marker candidates; the prototype developmental phase, in which immunological assays are established and validated in defined sample collectives; and finally the product development phase, with assay formats suitable for automated platforms. The hurdles that a potential candidate marker has to pass in each developmental phase before reaching the market are considerable. The costs are increasing from phase to phase, and in industry a number of questions concerning the medical need and the potential return on investment have to be answered before a proteomics discovery project is started. In this review, we will cover aspects of all three developmental phases including the repertoire of discovery tools for protein separation as well as giving an outline of modern principles of mass spectrometry for the identification of proteins.     

Microarray Data Analysis

The analysis of differential gene expression in microarray experiments requires the development of adequate statistical tools. This article describes a simple statistical method for detecting differential expression between two conditions with a low number of replicates. When comparing two group means using a traditional t-test, gene-specific variance estimates are unstable and can lead to wrong conclusions. We construct a likelihood ratio test while modelling these variances hierarchically across all genes, and express it as a t-test statistic. By borrowing information across genes we can take advantage of their large numbers, and still yield a gene-specific test statistic. We show that this hierarchical t-test is more powerful than its traditional version and generates less false positives in a simulation study, especially with small sample sizes. This approach can be extended to cases where there are more than two groups.     

Case-control studies of association in structured or admixed populations

Case-control tests for association are an important tool for mapping complex-trait genes. But population structure can invalidate this approach, leading to apparent associations at markers that are unlinked to disease loci. Family-based tests of association can avoid this problem, but such studies are often more expensive and in some cases--particularly for late-onset diseases--are impractical. In this review article we describe a series of approaches published over the past 2 years which use multilocus genotype data to enable valid case-control tests of association, even in the presence of population structure. These tests can be classified into two categories. "Genomic control" methods use the independent marker loci to adjust the distribution of a standard test statistic, while "structured association" methods infer the details of population structure en route to testing for association. We discuss the statistical issues involved in the different approaches and present results from simulations comparing the relative performance of the methods under a range of models.     

Development of diagnostic microsatellite markers from whole‐genome sequences of Ammodramus sparrows for assessing admixture in a hybrid zone

Studies of hybridization and introgression and, in particular, the identification of admixed individuals in natural populations benefit from the use of diagnostic genetic markers that reliably differentiate pure species from each other and their hybrid forms. Such diagnostic markers are often infrequent in the genomes of closely related species, and genomewide data facilitate their discovery. We used whole‐genome data from Illumina HiSeqS2000 sequencing of two recently diverged (600,000 years) and hybridizing, avian, sister species, the Saltmarsh (Ammodramus caudacutus) and Nelson's (A. nelsoni) Sparrow, to develop a suite of diagnostic markers for high‐resolution identification of pure and admixed individuals. We compared the microsatellite repeat regions identified in the genomes of the two species and selected a subset of 37 loci that differed between the species in repeat number. We screened these loci on 12 pure individuals of each species and report on the 34 that successfully amplified. From these, we developed a panel of the 12 most diagnostic loci, which we evaluated on 96 individuals, including individuals from both allopatric populations and sympatric individuals from the hybrid zone. Using simulations, we evaluated the power of the marker panel for accurate assignments of individuals to their appropriate pure species and hybrid genotypic classes (F1, F2, and backcrosses). The markers proved highly informative for species discrimination and had high accuracy for classifying admixed individuals into their genotypic classes. These markers will aid future investigations of introgressive hybridization in this system and aid conservation efforts aimed at monitoring and preserving pure species. Our approach is transferable to other study systems consisting of closely related and incipient species.     

Unconditional Exact Tests for Equivalence or Noninferiority for Paired Binary Endpoints

Problems of establishing equivalence or noninferiority between two medical diagnostic procedures involve comparisons of the response rates between correlated proportions. When the sample size is small, the asymptotic tests may not be reliable. This article proposes an unconditional exact test procedure to assess equivalence or noninferiority. Two statistics, a sample-based test statistic and a restricted maximum likelihood estimation (RMLE)-based test statistic, to define the rejection region of the exact test are considered. We show the p-value of the proposed unconditional exact tests can be attained at the boundary point of the null hypothesis. Assessment of equivalence is often based on a comparison of the confidence limits with the equivalence limits. We also derive the unconditional exact confidence intervals on the difference of the two proportion means for the two test statistics. A typical data set of comparing two diagnostic procedures is analyzed using the proposed unconditional exact and asymptotic methods. The p-value from the unconditional exact tests is generally larger than the p-value from the asymptotic tests. In other words, an exact confidence interval is generally wider than the confidence interval obtained from an asymptotic test.     

Nonparametric and semiparametric group sequential methods for comparing accuracy of diagnostic tests

SUMMARY: Comparison of the accuracy of two diagnostic tests using the receiver operating characteristic (ROC) curves from two diagnostic tests has been typically conducted using fixed sample designs. On the other hand, the human experimentation inherent in a comparison of diagnostic modalities argues for periodic monitoring of the accruing data to address many issues related to the ethics and efficiency of the medical study. To date, very little research has been done on the use of sequential sampling plans for comparative ROC studies, even when these studies may use expensive and unsafe diagnostic procedures. In this article we propose a nonparametric group sequential design plan. The nonparametric sequential method adapts a nonparametric family of weighted area under the ROC curve statistics (Wieand et al., 1989, Biometrika 76, 585-592) and a group sequential sampling plan. We illustrate the implementation of this nonparametric approach for sequentially comparing ROC curves in the context of diagnostic screening for nonsmall-cell lung cancer. We also describe a semiparametric sequential method based on proportional hazard models. We compare the statistical properties of the nonparametric approach with alternative semiparametric and parametric analyses in simulation studies. The results show the nonparametric approach is robust to model misspecification and has excellent finite-sample performance.