Improved nonparametric estimation of the optimal diagnostic cut‐off point associated with the Youden index under different sampling schemes

A diagnostic cut‐off point of a biomarker measurement is needed for classifying a random subject to be either diseased or healthy. However, the cut‐off point is usually unknown and needs to be estimated by some optimization criteria. One important criterion is the Youden index, which has been widely adopted in practice. The Youden index, which is defined as the maximum of (sensitivity + specificity ?1), directly measures the largest total diagnostic accuracy a biomarker can achieve. Therefore, it is desirable to estimate the optimal cut‐off point associated with the Youden index. Sometimes, taking the actual measurements of a biomarker is very difficult and expensive, while ranking them without the actual measurement can be relatively easy. In such cases, ranked set sampling can give more precise estimation than simple random sampling, as ranked set samples are more likely to span the full range of the population. In this study, kernel density estimation is utilized to numerically solve for an estimate of the optimal cut‐off point. The asymptotic distributions of the kernel estimators based on two sampling schemes are derived analytically and we prove that the estimators based on ranked set sampling are relatively more efficient than that of simple random sampling and both estimators are asymptotically unbiased. Furthermore, the asymptotic confidence intervals are derived. Intensive simulations are carried out to compare the proposed method using ranked set sampling with simple random sampling, with the proposed method outperforming simple random sampling in all cases. A real data set is analyzed for illustrating the proposed method.     

Youden Index and optimal cut-point estimated from observations affected by a lower limit of detection

The receiver operating characteristic (ROC) curve is used to evaluate a biomarker's ability for classifying disease status. The Youden Index (J), the maximum potential effectiveness of a biomarker, is a common summary measure of the ROC curve. In biomarker development, levels may be unquantifiable below a limit of detection (LOD) and missing from the overall dataset. Disregarding these observations may negatively bias the ROC curve and thus J. Several correction methods have been suggested for mean estimation and testing; however, little has been written about the ROC curve or its summary measures. We adapt non-parametric (empirical) and semi-parametric (ROC-GLM [generalized linear model]) methods and propose parametric methods (maximum likelihood (ML)) to estimate J and the optimal cut-point (c *) for a biomarker affected by a LOD. We develop unbiased estimators of J and c * via ML for normally and gamma distributed biomarkers. Alpha level confidence intervals are proposed using delta and bootstrap methods for the ML, semi-parametric, and non-parametric approaches respectively. Simulation studies are conducted over a range of distributional scenarios and sample sizes evaluating estimators' bias, root-mean square error, and coverage probability; the average bias was less than one percent for ML and GLM methods across scenarios and decreases with increased sample size. An example using polychlorinated biphenyl levels to classify women with and without endometriosis illustrates the potential benefits of these methods. We address the limitations and usefulness of each method in order to give researchers guidance in constructing appropriate estimates of biomarkers' true discriminating capabilities.     

Nonparametric confidence regions for the symmetry point-based optimal cutpoint and associated sensitivity of a continuous-scale diagnostic test

In medical research, diagnostic tests with continuous values are widely employed to attempt to distinguish between diseased and non-diseased subjects. The diagnostic accuracy of a test (or a biomarker) can be assessed by using the receiver operating characteristic (ROC) curve of the test. To summarize the ROC curve and primarily to determine an “optimal” threshold for test results to use in practice, several approaches may be considered, such as those based on the Youden index, on the so-called close-to-(0,1) point, on the concordance probability and on the symmetry point. In this paper, we focus on the symmetry point-based approach, that simultaneously controls the probabilities of the two types of correct classifications (healthy as healthy and diseased as diseased), and show how to get joint nonparametric confidence regions for the corresponding optimal cutpoint and the associated sensitivity (= specificity) value. Extensive simulation experiments are conducted to evaluate the finite sample performances of the proposed method. Real datasets are also used to illustrate its application.     

Estimation of the Youden Index and its associated cutoff point

The Youden Index is a frequently used summary measure of the ROC (Receiver Operating Characteristic) curve. It both, measures the effectiveness of a diagnostic marker and enables the selection of an optimal threshold value (cutoff point) for the marker. In this paper we compare several estimation procedures for the Youden Index and its associated cutoff point. These are based on (1) normal assumptions; (2) transformations to normality; (3) the empirical distribution function; (4) kernel smoothing. These are compared in terms of bias and root mean square error in a large variety of scenarios by means of an extensive simulation study. We find that the empirical method which is the most commonly used has the overall worst performance. In the estimation of the Youden Index the kernel is generally the best unless the data can be well transformed to achieve normality whereas in estimation of the optimal threshold value results are more variable.     

Construction of confidence intervals for the maximum of the Youden index and the corresponding cutoff point of a continuous biomarker

Evaluation of the overall accuracy of biomarkers might be based on average measures of the sensitivity for all possible specificities ‐and vice versa‐ or equivalently the area under the receiver operating characteristic (ROC) curve that is typically used in such settings. In practice clinicians are in need of a cutoff point to determine whether intervention is required after establishing the utility of a continuous biomarker. The Youden index can serve both purposes as an overall index of a biomarker's accuracy, that also corresponds to an optimal, in terms of maximizing the Youden index, cutoff point that in turn can be utilized for decision making. In this paper, we provide new methods for constructing confidence intervals for both the Youden index and its corresponding cutoff point. We explore approaches based on the delta approximation under the normality assumption, as well as power transformations to normality and nonparametric kernel‐ and spline‐based approaches. We compare our methods to existing techniques through simulations in terms of coverage and width. We then apply the proposed methods to serum‐based markers of a prospective observational study involving diagnosis of late‐onset sepsis in neonates.     

Serial determination of CEA and CA 15.3 in breast cancer follow-up: An assessment of their diagnostic accuracy for the detection of tumour recurrences

We studied the diagnostic accuracy of carcinoembryonic antigen (CEA) and cancer antigen 15.3 (CA 15.3) in detecting breast cancer recurrence. Biomarker follow-up determinations, made over 900 patients, were related to local–regional or distant recurrence using statistical models for longitudinal data. The diagnostic accuracy was quantified in terms of sensitivity, specificity and Youden index. The biomarkers were poorly predictive of local–regional recurrence. As for distant recurrence, the best diagnostic accuracy was obtained considering the two biomarkers jointly and combining two positivity criteria: a value above the normal limit or a difference between two consecutive measurements greater than the critical difference for at least one biomarker. A third criterion, based on within-patient comparison between follow-up determinations and a baseline, failed to improve the above result. CEA and CA 15.3 might play a role in patient monitoring during follow-up for the search of distant recurrence.     

Confidence interval estimation of the difference between two sensitivities to the early disease stage

Although most of the statistical methods for diagnostic studies focus on disease processes with binary disease status, many diseases can be naturally classified into three ordinal diagnostic categories, that is normal, early stage, and fully diseased. For such diseases, the volume under the ROC surface (VUS) is the most commonly used index of diagnostic accuracy. Because the early disease stage is most likely the optimal time window for therapeutic intervention, the sensitivity to the early diseased stage has been suggested as another diagnostic measure. For the purpose of comparing the diagnostic abilities on early disease detection between two markers, it is of interest to estimate the confidence interval of the difference between sensitivities to the early diseased stage. In this paper, we present both parametric and non‐parametric methods for this purpose. An extensive simulation study is carried out for a variety of settings for the purpose of evaluating and comparing the performance of the proposed methods. A real example of Alzheimer's disease (AD) is analyzed using the proposed approaches.     

Estimation of the Optimal Statistical Quality Control Sampling Time Intervals Using a Residual Risk Measure

Background

An open problem in clinical chemistry is the estimation of the optimal sampling time intervals for the application of statistical quality control (QC) procedures that are based on the measurement of control materials. This is a probabilistic risk assessment problem that requires reliability analysis of the analytical system, and the estimation of the risk caused by the measurement error.

Methodology/Principal Findings

Assuming that the states of the analytical system are the reliability state, the maintenance state, the critical-failure modes and their combinations, we can define risk functions based on the mean time of the states, their measurement error and the medically acceptable measurement error. Consequently, a residual risk measure rr can be defined for each sampling time interval. The rr depends on the state probability vectors of the analytical system, the state transition probability matrices before and after each application of the QC procedure and the state mean time matrices. As optimal sampling time intervals can be defined those minimizing a QC related cost measure while the rr is acceptable. I developed an algorithm that estimates the rr for any QC sampling time interval of a QC procedure applied to analytical systems with an arbitrary number of critical-failure modes, assuming any failure time and measurement error probability density function for each mode. Furthermore, given the acceptable rr, it can estimate the optimal QC sampling time intervals.

Conclusions/Significance

It is possible to rationally estimate the optimal QC sampling time intervals of an analytical system to sustain an acceptable residual risk with the minimum QC related cost. For the optimization the reliability analysis of the analytical system and the risk analysis of the measurement error are needed.     

Classifying genealogical origins in hybrid populations using dominant markers

In hybrid studies, potential for error is high when classifying genealogical origins of individuals (e.g., parental, F1, F2) based on their genotypic arrays. For codominant markers, previous researchers have considered the probability of misclassification by genotypic inspection and proposed alternative maximum-likelihood approaches to estimating genealogical class frequencies. Recently developed dominant marker systems may significantly increase the number of diagnostic loci available for hybrid studies. I examine probabilities of classification error based on the number of dominant loci. As in earlier studies, I assume that only parental and first- and second-generation hybrid crosses between two taxa potentially exist. Thirteen loci with dominant expression from each parental taxon (i.e., 26 total loci) are needed to reduce classification error below 5% for F2 individuals, compared to 13 codominant loci for the same error rate. Use of loci in similar numbers from both taxa most efficiently increases power to characterize all genealogical classes. In contrast, classification of backcrosses to one parental taxon is wholly dependent on loci from the other taxon. Use of dominant diagnostic markers may increase the power and expand the use of maximum-likelihood methods for evaluating hybrid mixtures.     

A Two-stage Procedure with Controlled Error Probabilities for Testing Bioequivalence

The procedures currently used for testing the bioequivalence of two drug formulations achieve control over the error probability of erroneously accepting bioequivalence or over the probability of erroneous rejection, but not over both error probabilities. A two-stage procedure that rectifies this drawback is presented, assuming that the performance of the drug is characterized by a normally distributed variate.     

The optimal ratio of cases to controls for estimating the classification accuracy of a biomarker

The case-control design is frequently used to study the discriminatory accuracy of a screening or diagnostic biomarker. Yet, the appropriate ratio in which to sample cases and controls has never been determined. It is common for researchers to sample equal numbers of cases and controls, a strategy that can be optimal for studies of association. However, considerations are quite different when the biomarker is to be used for classification. In this paper, we provide an expression for the optimal case-control ratio, when the accuracy of the biomarker is quantified by the receiver operating characteristic (ROC) curve. We show how it can be integrated with choosing the overall sample size to yield an efficient study design with specified power and type-I error. We also derive the optimal case-control ratios for estimating the area under the ROC curve and the area under part of the ROC curve. Our methods are applied to a study of a new marker for adenocarcinoma in patients with Barrett's esophagus.     

Individual variation in early‐life telomere length and survival in a wild mammal

Individual variation in survival probability due to differential responses to early‐life environmental conditions is important in the evolution of life histories and senescence. A biomarker allowing quantification of such individual variation, and which links early‐life environmental conditions with survival by providing a measure of conditions experienced, is telomere length. Here, we examined telomere dynamics among 24 cohorts of European badgers (Meles meles). We found a complex cross‐sectional relationship between telomere length and age, with no apparent loss over the first 29 months, but with both decreases and increases in telomere length at older ages. Overall, we found low within‐individual consistency in telomere length across individual lifetimes. Importantly, we also observed increases in telomere length within individuals, which could not be explained by measurement error alone. We found no significant sex differences in telomere length, and provide evidence that early‐life telomere length predicts lifespan. However, while early‐life telomere length predicted survival to adulthood (≥1 year old), early‐life telomere length did not predict adult survival probability. Furthermore, adult telomere length did not predict survival to the subsequent year. These results show that the relationship between early‐life telomere length and lifespan was driven by conditions in early‐life, where early‐life telomere length varied strongly among cohorts. Our data provide evidence for associations between early‐life telomere length and individual life history, and highlight the dynamics of telomere length across individual lifetimes due to individuals experiencing different early‐life environments.     

Stochastic Dynamic Model for Estimation of Rate Constants and Their Variances from Noisy and Heterogeneous PET Measurements

Tissue heterogeneity, radioactive decay and measurement noise are the main error sources in compartmental modeling used to estimate the physiologic rate constants of various radiopharmaceuticals from a dynamic PET study. We introduce a new approach to this problem by modeling the tissue heterogeneity with random rate constants in compartment models. In addition, the Poisson nature of the radioactive decay is included as a Poisson random variable in the measurement equations. The estimation problem will be carried out using the maximum likelihood estimation. With this approach, we do not only get accurate mean estimates for the rate constants, but also estimates for tissue heterogeneity within the region of interest and other possibly unknown model parameters, e.g. instrument noise variance, as well. We also avoid the problem of the optimal weighting of the data related to the conventionally used weighted least-squares method. The new approach was tested with simulated time–activity curves from the conventional three compartment – three rate constants model with normally distributed rate constants and with a noise mixture of Poisson and normally distributed random variables. Our simulation results showed that this new model gave accurate estimates for the mean of the rate constants, the measurement noise parameter and also for the tissue heterogeneity, i.e. for the variance of the rate constants within the region of interest.     

Confidence regions for treatment effects in subgroups in biomarker stratified designs

Subgroup analysis has important applications in the analysis of controlled clinical trials. Sometimes the result of the overall group fails to demonstrate that the new treatment is better than the control therapy, but for a subgroup of patients, the treatment benefit may exist; or sometimes, the new treatment is better for the overall group but not for a subgroup. Hence we are interested in constructing a simultaneous confidence interval for the difference of the treatment effects in a subgroup and the overall group. Subgroups are usually formed on the basis of a predictive biomarker such as age, sex, or some genetic marker. While, for example, age can be detected precisely, it is often only possible to detect the biomarker status with a certain probability. Because patients detected with a positive or negative biomarker may not be truly biomarker positive or negative, responses in the subgroups depend on the treatment therapy as well as on the sensitivity and specificity of the assay used in detecting the biomarkers. In this work, we show how (approximate) simultaneous confidence intervals and confidence ellipsoid for the treatment effects in subgroups can be found for biomarker stratified clinical trials using a normal framework with normally distributed or binary data. We show that these intervals maintain the nominal confidence level via simulations.     

A repeated measures approach to pooled and calibrated biomarker data

Participant-level meta-analysis across multiple studies increases the sample size for pooled analyses, thereby improving precision in effect estimates and enabling subgroup analyses. For analyses involving biomarker measurements as an exposure of interest, investigators must first calibrate the data to address measurement variability arising from usage of different laboratories and/or assays. In practice, the calibration process involves reassaying a random subset of biospecimens from each study at a central laboratory and fitting models that relate the study-specific “local” and central laboratory measurements. Previous work in this area treats the calibration process from the perspective of measurement error techniques and imputes the estimated central laboratory value among individuals with only a local laboratory measurement. In this work, we propose a repeated measures method to calibrate biomarker measurements pooled from multiple studies with study-specific calibration subsets. We account for correlation between measurements made on the same person and between measurements made at the same laboratory. We demonstrate that the repeated measures approach provides valid inference, and compare it to existing calibration approaches grounded in measurement error techniques in an example describing the association between circulating vitamin D and stroke.     

Estimation of the diagnostic threshold accounting for decision costs and sampling uncertainty

Medical diagnostic tests are used to classify subjects as non-diseased or diseased. The classification rule usually consists of classifying subjects using the values of a continuous marker that is dichotomised by means of a threshold. Here, the optimum threshold estimate is found by minimising a cost function that accounts for both decision costs and sampling uncertainty. The cost function is optimised either analytically in a normal distribution setting or empirically in a free-distribution setting when the underlying probability distributions of diseased and non-diseased subjects are unknown. Inference of the threshold estimates is based on approximate analytically standard errors and bootstrap-based approaches. The performance of the proposed methodology is assessed by means of a simulation study, and the sample size required for a given confidence interval precision and sample size ratio is also calculated. Finally, a case example based on previously published data concerning the diagnosis of Alzheimer's patients is provided in order to illustrate the procedure.     

Partial AUC estimation and regression

Accurate diagnosis of disease is a critical part of health care. New diagnostic and screening tests must be evaluated based on their abilities to discriminate diseased from nondiseased states. The partial area under the receiver operating characteristic (ROC) curve is a measure of diagnostic test accuracy. We present an interpretation of the partial area under the curve (AUC), which gives rise to a nonparametric estimator. This estimator is more robust than existing estimators, which make parametric assumptions. We show that the robustness is gained with only a moderate loss in efficiency. We describe a regression modeling framework for making inference about covariate effects on the partial AUC. Such models can refine knowledge about test accuracy. Model parameters can be estimated using binary regression methods. We use the regression framework to compare two prostate-specific antigen biomarkers and to evaluate the dependence of biomarker accuracy on the time prior to clinical diagnosis of prostate cancer.     

Multiple comparisons in the randomization analysis of designed experiments with growth curve responses

A randomization approach to multiple comparisons is developed for comparing several growth curves in randomized experiments. The exact Type I probability error rate for these comparisons may be prespecified, and a Type I error probability for each component test can be evaluated. These procedures are free of many of the standard assumptions for analyzing growth curves and for making multiple comparisons. An application of the procedure gives all pairwise comparisons among the mean growth curves associated with four treatments in an animal experiment using a Youden square design, where growth curves are obtained on monitoring hormone levels over time.     

Atypical indices in human paleontology

For a normally distributed variable, the index of atypicality associated with a measurement of that variable is defined as the probability of finding a result closer to the mean of the reference population than the one actually observed. The method extends straight forwardly to multivariate situations, thus providing a joint interpretation of multiple observations recorded on the same subject. In human palaeontology, atypicality indices allow to analyze and to compare the results obtained from variables closely associated with specific traits of the morphology. We briefly outline the principle of the method and give a computer program to carry out the calculations. Finally, we apply the use of univariate and multivariate atypicality indices to data recorded on 53 Upper Palaeolithic skulls.