Evaluating the Predictive Value of Biomarkers with Stratified Case‐Cohort Design

Summary Identification of novel biomarkers for risk assessment is important for both effective disease prevention and optimal treatment recommendation. Discovery relies on the precious yet limited resource of stored biological samples from large prospective cohort studies. Case‐cohort sampling design provides a cost‐effective tool in the context of biomarker evaluation, especially when the clinical condition of interest is rare. Existing statistical methods focus on making efficient inference on relative hazard parameters from the Cox regression model. Drawing on recent theoretical development on the weighted likelihood for semiparametric models under two‐phase studies ( Breslow and Wellner, 2007 ), we propose statistical methods to evaluate accuracy and predictiveness of a risk prediction biomarker, with censored time‐to‐event outcome under stratified case‐cohort sampling. We consider nonparametric methods and a semiparametric method. We derive large sample properties of proposed estimators and evaluate their finite sample performance using numerical studies. We illustrate new procedures using data from Framingham Offspring Study to evaluate the accuracy of a recently developed risk score incorporating biomarker information for predicting cardiovascular disease.     

Binary regression: Total gain in positive and negative predictive values

Models that predict disease incidence or disease recurrence are attractive for clinicians as well as for patients. The usefulness of a risk prediction model is linked to the two questions whether the observed outcome is confirmed by the prediction and whether the risk prediction is accurate in predicting the future outcome, respectively. The first phrasing of the question is linked to considering sensitivity and specificity and the latter to the positive and negative predictive values. We present the measures of standardized total gain in positive and negative predictive values dealing with the performance or accuracy of the prediction model for a binary outcome. Both measures provide a useful tool for assessing the performance or accuracy of a set of predictor variables for the prediction of a binary outcome. This concept is a tool for evaluating the optimal prediction model in future research.     

On Being Careful What We Wish for: Some Difficulties with Operationalizing the Precautionary Principle

An important part of the Precautionary Principle is that taking action is justified for protecting public health even when there is some scientific uncertainty. We examine here the two components of this central feature of the precautionary principle, scientific uncertainty and decision making. In order to operationalize the principle we should examine the consequences of its decision rules and how they perform under various conditions. The performance of decision rules in disease screening is measured by the sensitivity and specificity of the rule, but the consequences for the patient are given by the positive and negative predictive values, determined not only by the performance of the rule by the prevalence of the disease in the population. We look at positive and negative predictive value of the Precautionary Principle from the standopoint of the costs to one or other parts of society, show that the usual rule which tends to maximize sensitivity in favor of specificity may have unexpected consequences, and demonstrate that it is sometimes possible to trade sensitivity and specificity off against each other in a way the improves both positive and negative predictive value, or worse, degrades both.We conclude by asking if certain strategies are better for one or the other kinds of uncertainty.     

Bayesian personalized treatment selection strategies that integrate predictive with prognostic determinants

The evolution of “informatics” technologies has the potential to generate massive databases, but the extent to which personalized medicine may be effectuated depends on the extent to which these rich databases may be utilized to advance understanding of the disease molecular profiles and ultimately integrated for treatment selection, necessitating robust methodology for dimension reduction. Yet, statistical methods proposed to address challenges arising with the high‐dimensionality of omics‐type data predominately rely on linear models and emphasize associations deriving from prognostic biomarkers. Existing methods are often limited for discovering predictive biomarkers that interact with treatment and fail to elucidate the predictive power of their resultant selection rules. In this article, we present a Bayesian predictive method for personalized treatment selection that is devised to integrate both the treatment predictive and disease prognostic characteristics of a particular patient's disease. The method appropriately characterizes the structural constraints inherent to prognostic and predictive biomarkers, and hence properly utilizes these complementary sources of information for treatment selection. The methodology is illustrated through a case study of lower grade glioma. Theoretical considerations are explored to demonstrate the manner in which treatment selection is impacted by prognostic features. Additionally, simulations based on an actual leukemia study are provided to ascertain the method's performance with respect to selection rules derived from competing methods.     

Evaluating prognostic accuracy of biomarkers under competing risk

To develop more targeted intervention strategies, an important research goal is to identify markers predictive of clinical events. A crucial step toward this goal is to characterize the clinical performance of a marker for predicting different types of events. In this article, we present statistical methods for evaluating the performance of a prognostic marker in predicting multiple competing events. To capture the potential time-varying predictive performance of the marker and incorporate competing risks, we define time- and cause-specific accuracy summaries by stratifying cases based on causes of failure. Such definition would allow one to evaluate the predictive accuracy of a marker for each type of event and compare its predictiveness across event types. Extending the nonparametric crude cause-specific receiver operating characteristics curve estimators by Saha and Heagerty (2010), we develop inference procedures for a range of cause-specific accuracy summaries. To estimate the accuracy measures and assess how covariates may affect the accuracy of a marker under the competing risk setting, we consider two forms of semiparametric models through the cause-specific hazard framework. These approaches enable a flexible modeling of the relationships between the marker and failure times for each cause, while efficiently accommodating additional covariates. We investigate the asymptotic property of the proposed accuracy estimators and demonstrate the finite sample performance of these estimators through simulation studies. The proposed procedures are illustrated with data from a prostate cancer prognostic study.     

Empirical modelling of lake water-temperature relationships: a comparison of approaches

1.  As a result of the role that temperature plays in many aquatic processes, good predictive models of annual maximum near-surface lake water temperature across large spatial scales are needed, particularly given concerns regarding climate change. Comparisons of suitable modelling approaches are required to determine their relative merit and suitability for providing good predictions of current conditions. We developed models predicting annual maximum near-surface lake water temperatures for lakes across Canada using four statistical approaches: multiple regression, regression tree, artificial neural networks and Bayesian multiple regression.
2.  Annual maximum near-surface (from 0 to 2 m) lake water-temperature data were obtained for more than 13 000 lakes and were matched to geographic, climatic, lake morphology, physical habitat and water chemistry data. We modelled 2348 lakes and three subsets thereof encompassing different spatial scales and predictor variables to identify the relative importance of these variables at predicting lake temperature.
3.  Although artificial neural networks were marginally better for three of the four data sets, multiple regression was considered to provide the best solution based on the combination of model performance and computational complexity. Climatic variables and date of sampling were the most important variables for predicting water temperature in our models.
4.  Lake morphology did not play a substantial role in predicting lake temperature across any of the spatial scales. Maximum near-surface temperatures for Canadian lakes appeared to be dominated by large-scale climatic and geographic patterns, rather than lake-specific variables, such as lake morphology and water chemistry.     

Comparison of SurePath® and ThinPrep® liquid‐based cervical cytology using positive predictive value,atypical predictive value and total predictive value as performance indicators

P. K. Wright, J. Marshall and M. Desai Comparison of SurePath ^® and ThinPrep ^® liquid‐based cervical cytology using positive predictive value, atypical predictive value and total predictive value as performance indicators Objective: Two liquid‐based cytology (LBC) systems are in widespread use in the UK: ThinPrep^® and SurePath^®. A number of studies have now compared LBC with conventional cytology in cervical screening. However, to date, we are aware of no studies that have compared ThinPrep^® with SurePath^® LBC. As the selection and use of specific diagnostic systems in a laboratory has significant clinical and economic implications, there is a clear need to compare directly existing LBC technology. The objective of this study was to compare ThinPrep^® with SurePath^® LBC in a single cytology laboratory using performance indicators. Methods: Data were collected for all cervical cytology samples processed at Manchester Cytology Centre over a 1‐year period. ThinPrep^® LBC was compared with SurePath^® LBC using positive predictive value (PPV), atypical predictive value (APV) and total predictive value (TPV), reflecting outcome of cervical intraepithelial neoplasia (CIN) grade 2 or worse for high‐grade dyskaryosis (PPV), low‐grade dyskaryosis or borderline (atypical) cytology (APV) and all (total) abnormal cytology (TPV). Results: 2287 (out of 56 467) (ThinPrep^®) and 586 (out of 22 824) (SurePath^®) samples showed borderline or worse cytology after exclusion criteria. PPV, APV and TPV were within acceptable ranges for both ThinPrep^® and SurePath^®. Conclusions: ThinPrep^® and SurePath^® were equivalent based on three performance indicators. We suggest that APV and TPV should be used as an adjunct to PPV and other methods of quality assurance for cervical screening.     

Quantifying and comparing the predictive accuracy of continuous prognostic factors for binary outcomes

The positive and negative predictive values are standard ways of quantifying predictive accuracy when both the outcome and the prognostic factor are binary. Methods for comparing the predictive values of two or more binary factors have been discussed previously (Leisenring et al., 2000, Biometrics 56, 345-351). We propose extending the standard definitions of the predictive values to accommodate prognostic factors that are measured on a continuous scale and suggest a corresponding graphical method to summarize predictive accuracy. Drawing on the work of Leisenring et al. we make use of a marginal regression framework and discuss methods for estimating these predictive value functions and their differences within this framework. The methods presented in this paper have the potential to be useful in a number of areas including the design of clinical trials and health policy analysis.     

Evaluation of a mathematical model structure describing the effect of (gel) structure on the growth of Listeria innocua, Lactococcus lactis and Salmonella Typhimurium

Aims:  To evaluate a novel secondary model structure ( Int J Food Microbiol 2008; 128: 67) that describes the effect of medium structure on the maximum specific growth rate ( μ _max) of Salmonella Typhimurium on the growth of S. Typhimurium, Listeria innocua , Lactococcus lactis and Listeria monocytogenes .
Methods and Results:  In the present study, the novel secondary model is validated for S . Typhimurium in more realistic media, namely, pasteurized milk and a cheese mimicking medium. The predictions were accurate. Next, the secondary model structure was evaluated in a two step and a global regression procedure on literature data. On the one hand, the growth of two other micro-organisms, namely L. innocua and L. lactis , in monoculture for varying gelatine concentrations was tested and on the other hand the growth rate of L. monocytogenes was fitted in a broth of which the viscosity was altered with polyvinylpyrrolidone. The model was able to describe the effect of increasing gelatine concentration or viscosity accurately.
Conclusions:  The proposed secondary model structure is able to describe the effect of gelatine concentration on the μ _max of the micro-organisms tested in this study.
Significance and Impact of the Study:  In predictive microbiology, much attention has been paid to the effect of food structure on the μ _max of bacteria. However, to the authors' knowledge, a lack of secondary models still exists to describe this effect. Although the proposed model is empirical, the model parameters have clear biological meaning. The predictive power of the model to describe the effect of food structure is clearly illustrated.     

A comparison of statistical methods for the detection of hepatocellular carcinoma based on serum biomarkers and clinical variables

Background

Currently, a surgical approach is the best curative treatment for those with hepatocellular carcinoma (HCC). However, this requires HCC detection and removal of the lesion at an early stage. Unfortunately, most cases of HCC are detected at an advanced stage because of the lack of accurate biomarkers that can be used in the surveillance of those at risk. It is believed that biomarkers that could detect HCC early will play an important role in the successful treatment of HCC.

Methods

In this study, we analyzed serum levels of alpha fetoprotein, Golgi protein, fucosylated alpha-1-anti-trypsin, and fucosylated kininogen from 113 patients with cirrhosis and 164 serum samples from patients with cirrhosis plus HCC. We utilized two different methods, namely, stepwise penalized logistic regression (stepPLR) and model-based classification and regression trees (mob), along with the inclusion of clinical and demographic factors such as age and gender, to determine if these improved algorithms could be used to increase the detection of cancer.

Results and discussion

The performance of multiple biomarkers was found to be better than that of individual biomarkers. Using several statistical methods, we were able to detect HCC in the background of cirrhosis with an area under the receiver operating characteristic curve of at least 0.95. stepPLR and mob demonstrated better predictive performance relative to logistic regression (LR), penalized LR and classification and regression trees (CART) used in our prior study based on three-fold cross-validation and leave one out cross-validation. In addition, mob provided unparalleled intuitive interpretation of results and potential cut-points for biomarker levels. The inclusion of age and gender improved the overall performance of both methods among all models considered, while the stratified male-only subset provided the best overall performance among all methods and models considered.

Conclusions

In addition to multiple biomarkers, the incorporation of age and gender into statistical models significantly improved their predictive performance in the detection of HCC.

     

Translation of proteomic biomarkers into FDA approved cancer diagnostics: issues and challenges

Tremendous efforts have been made over the past few decades to discover novel cancer biomarkers for use in clinical practice. However, a striking discrepancy exists between the effort directed toward biomarker discovery and the number of markers that make it into clinical practice. One of the confounding issues in translating a novel discovery into clinical practice is that quite often the scientists working on biomarker discovery have limited knowledge of the analytical, diagnostic, and regulatory requirements for a clinical assay. This review provides an introduction to such considerations with the aim of generating more extensive discussion for study design, assay performance, and regulatory approval in the process of translating new proteomic biomarkers from discovery into cancer diagnostics. We first describe the analytical requirements for a robust clinical biomarker assay, including concepts of precision, trueness, specificity and analytical interference, and carryover. We next introduce the clinical considerations of diagnostic accuracy, receiver operating characteristic analysis, positive and negative predictive values, and clinical utility. We finish the review by describing components of the FDA approval process for protein-based biomarkers, including classification of biomarker assays as medical devices, analytical and clinical performance requirements, and the approval process workflow. While we recognize that the road from biomarker discovery, validation, and regulatory approval to the translation into the clinical setting could be long and difficult, the reward for patients, clinicians and scientists could be rather significant.     

Efficient screening of predictive biomarkers for individual treatment selection

The development of molecular diagnostic tools to achieve individualized medicine requires identifying predictive biomarkers associated with subgroups of individuals who might receive beneficial or harmful effects from different available treatments. However, due to the large number of candidate biomarkers in the large‐scale genetic and molecular studies, and complex relationships among clinical outcome, biomarkers, and treatments, the ordinary statistical tests for the interactions between treatments and covariates have difficulties from their limited statistical powers. In this paper, we propose an efficient method for detecting predictive biomarkers. We employ weighted loss functions of Chen et al. to directly estimate individual treatment scores and propose synthetic posterior inference for effect sizes of biomarkers. We develop an empirical Bayes approach, namely, we estimate unknown hyperparameters in the prior distribution based on data. We then provide efficient screening methods for the candidate biomarkers via optimal discovery procedure with adequate control of false discovery rate. The proposed method is demonstrated in simulation studies and an application to a breast cancer clinical study in which the proposed method was shown to detect the much larger numbers of significant biomarkers than existing standard methods.     

Covariate adjustment in continuous biomarker assessment

Continuous biomarkers are common for disease screening and diagnosis. To reach a dichotomous clinical decision, a threshold would be imposed to distinguish subjects with disease from nondiseased individuals. Among various performance metrics, specificity at a controlled sensitivity level (or vice versa) is often desirable because it directly targets the clinical utility of the intended clinical test. Meanwhile, covariates, such as age, race, as well as sample collection conditions, could impact the biomarker distribution and may also confound the association between biomarker and disease status. Therefore, covariate adjustment is important in such biomarker evaluation. Most existing covariate adjustment methods do not specifically target the desired sensitivity/specificity level, but rather do so for the entire biomarker distribution. As such, they might be more prone to model misspecification. In this paper, we suggest a parsimonious quantile regression model for the diseased population, only locally at the controlled sensitivity level, and assess specificity with covariate-specific control of the sensitivity. Variance estimates are obtained from a sample-based approach and bootstrap. Furthermore, our proposed local model extends readily to a global one for covariate adjustment for the receiver operating characteristic (ROC) curve over the sensitivity continuum. We demonstrate computational efficiency of this proposed method and restore the inherent monotonicity in the estimated covariate-adjusted ROC curve. The asymptotic properties of the proposed estimators are established. Simulation studies show favorable performance of the proposal. Finally, we illustrate our method in biomarker evaluation for aggressive prostate cancer.     

Using a graph of the abnormal predictive value versus the positive predictive value for the determination of outlier laboratories in the National Health Service cervical screening programme

R.G. Blanks Using a graph of the abnormal predictive value versus the positive predictive value for the determination of outlier laboratories in the National Health Service cervical screening programme. Objective: The positive predictive value (PPV) for the detection of cervical intraepithelial neoplasia (CIN) grade 2 or worse of referral to colposcopy from moderate dyskaryosis or worse (equivalent to high‐grade squamous intraepithelial lesion or worse) is a standard performance measure in the National Health Service cervical screening programme. The current target is to examine ‘outlier’ laboratories with PPVs outside the 10th–90th percentile, which automatically identifies 20% of laboratories for further investigation. A more targeted method of identifying outliers may be more useful. Methods: A similar measure to the PPV, the abnormal predictive value (APV), can be defined as the predictive value for CIN2 or worse for referrals from borderline (includes atypical squamous and glandular cells) and mild dyskaryosis (equivalent to low‐grade squamous intraepithelial lesion) combined. A scatter plot of the APV versus the PPV can be produced (the APV‐PPV diagram). Three kinds of ‘outlier’ can be defined on the diagram to help determine laboratories with unusual data. These are termed a true outlier value (TOV) or an extreme value (EV) for either PPV or APV, or a residual extreme value (REV) from the APV‐PPV best line of fit. Results: Using annual return information for 2007/8 from 124 laboratories, two were defined as having EVs for PPV (both had a relatively low PPV of 62%). For APV, four laboratories were considered to have EVs of 34%, 34%, 34% and 4% and one was considered to be a TO with an APV of 45%. Five were identified as REV laboratories, although three of these were also identified as having extreme or outlier values, leaving two that had not been identified by the other methods. A total of eight (6%) laboratories were therefore identified as meriting further investigation using this methodology. Conclusions: The method proposed could be a useful alternative to the current method of identifying outliers. Slide exchange studies between the identified laboratories, particularly those at opposing ends of the diagram, or other further investigations of such laboratories, may be instructive in understanding why such variation occurs, and could therefore potentially, lead to improvements in the national programme.     

Robust Estimation of Area Under ROC Curve Using Auxiliary Variables in the Presence of Missing Biomarker Values

Summary In medical research, the receiver operating characteristic (ROC) curves can be used to evaluate the performance of biomarkers for diagnosing diseases or predicting the risk of developing a disease in the future. The area under the ROC curve (ROC AUC), as a summary measure of ROC curves, is widely utilized, especially when comparing multiple ROC curves. In observational studies, the estimation of the AUC is often complicated by the presence of missing biomarker values, which means that the existing estimators of the AUC are potentially biased. In this article, we develop robust statistical methods for estimating the ROC AUC and the proposed methods use information from auxiliary variables that are potentially predictive of the missingness of the biomarkers or the missing biomarker values. We are particularly interested in auxiliary variables that are predictive of the missing biomarker values. In the case of missing at random (MAR), that is, missingness of biomarker values only depends on the observed data, our estimators have the attractive feature of being consistent if one correctly specifies, conditional on auxiliary variables and disease status, either the model for the probabilities of being missing or the model for the biomarker values. In the case of missing not at random (MNAR), that is, missingness may depend on the unobserved biomarker values, we propose a sensitivity analysis to assess the impact of MNAR on the estimation of the ROC AUC. The asymptotic properties of the proposed estimators are studied and their finite‐sample behaviors are evaluated in simulation studies. The methods are further illustrated using data from a study of maternal depression during pregnancy.     

Combining multiple biomarker models in logistic regression

Summary .   In medical research, there is great interest in developing methods for combining biomarkers. We argue that selection of markers should also be considered in the process. Traditional model/variable selection procedures ignore the underlying uncertainty after model selection. In this work, we propose a novel model-combining algorithm for classification in biomarker studies. It works by considering weighted combinations of various logistic regression models; five different weighting schemes are considered in the article. The weights and algorithm are justified using decision theory and risk-bound results. Simulation studies are performed to assess the finite-sample properties of the proposed model-combining method. It is illustrated with an application to data from an immunohistochemical study in prostate cancer.     

Regularized ROC method for disease classification and biomarker selection with microarray data

MOTIVATION: An important application of microarrays is to discover genomic biomarkers, among tens of thousands of genes assayed, for disease classification. Thus there is a need for developing statistical methods that can efficiently use such high-throughput genomic data, select biomarkers with discriminant power and construct classification rules. The ROC (receiver operator characteristic) technique has been widely used in disease classification with low-dimensional biomarkers because (1) it does not assume a parametric form of the class probability as required for example in the logistic regression method; (2) it accommodates case-control designs and (3) it allows treating false positives and false negatives differently. However, due to computational difficulties, the ROC-based classification has not been used with microarray data. Moreover, the standard ROC technique does not incorporate built-in biomarker selection. RESULTS: We propose a novel method for biomarker selection and classification using the ROC technique for microarray data. The proposed method uses a sigmoid approximation to the area under the ROC curve as the objective function for classification and the threshold gradient descent regularization method for estimation and biomarker selection. Tuning parameter selection based on the V-fold cross validation and predictive performance evaluation are also investigated. The proposed approach is demonstrated with a simulation study, the Colon data and the Estrogen data. The proposed approach yields parsimonious models with excellent classification performance.     

An overview of techniques for linking high-dimensional molecular data to time-to-event endpoints by risk prediction models

Analysis of molecular data promises identification of biomarkers for improving prognostic models, thus potentially enabling better patient management. For identifying such biomarkers, risk prediction models can be employed that link high-dimensional molecular covariate data to a clinical endpoint. In low-dimensional settings, a multitude of statistical techniques already exists for building such models, e.g. allowing for variable selection or for quantifying the added value of a new biomarker. We provide an overview of techniques for regularized estimation that transfer this toward high-dimensional settings, with a focus on models for time-to-event endpoints. Techniques for incorporating specific covariate structure are discussed, as well as techniques for dealing with more complex endpoints. Employing gene expression data from patients with diffuse large B-cell lymphoma, some typical modeling issues from low-dimensional settings are illustrated in a high-dimensional application. First, the performance of classical stepwise regression is compared to stage-wise regression, as implemented by a component-wise likelihood-based boosting approach. A second issues arises, when artificially transforming the response into a binary variable. The effects of the resulting loss of efficiency and potential bias in a high-dimensional setting are illustrated, and a link to competing risks models is provided. Finally, we discuss conditions for adequately quantifying the added value of high-dimensional gene expression measurements, both at the stage of model fitting and when performing evaluation.     

Prostate cancer outcome: epidemiology and biostatistics

Substantial gaps exist in our ability to accurately predict prognosis, and these gaps limit our understanding of the complex mechanisms that contribute to the greatest cancer epidemic of our time, prostate cancer. This review addresses contemporary epidemiologic and biostatistical issues in prostate cancer. It covers the science of outcome prediction and biomarker evaluation, recognition of the need to combine biomarkers to improve the accuracy of our outcome estimates and an analysis of current outcome assessment methods, including the TNM staging system and multivariate regression models. The simplicity and intuitive ease of the current TNM staging system must be balanced against its serious limitations in predictive accuracy and its loss of clinical utility. Statistical regression methods are required as we move to the new era of personalized medicine. We must implement statistical approaches that integrate the new molecular biomarkers with existing prognostic biomarkers to accurately predict which patients require treatment and to determine the optimal therapy.