Comparing biomarkers as principal surrogate endpoints

Recently a new definition of surrogate endpoint, the "principal surrogate," was proposed based on causal associations between treatment effects on the biomarker and on the clinical endpoint. Despite its appealing interpretation, limited research has been conducted to evaluate principal surrogates, and existing methods focus on risk models that consider a single biomarker. How to compare principal surrogate value of biomarkers or general risk models that consider multiple biomarkers remains an open research question. We propose to characterize a marker or risk model's principal surrogate value based on the distribution of risk difference between interventions. In addition, we propose a novel summary measure (the standardized total gain) that can be used to compare markers and to assess the incremental value of a new marker. We develop a semiparametric estimated-likelihood method to estimate the joint surrogate value of multiple biomarkers. This method accommodates two-phase sampling of biomarkers and is more widely applicable than existing nonparametric methods by incorporating continuous baseline covariates to predict the biomarker(s), and is more robust than existing parametric methods by leaving the error distribution of markers unspecified. The methodology is illustrated using a simulated example set and a real data set in the context of HIV vaccine trials.     

Commentary on "Principal stratification - a goal or a tool?" by Judea Pearl

This commentary takes up Pearl's welcome challenge to clearly articulate the scientific value of principal stratification estimands that we and colleagues have investigated, in the area of randomized placebo-controlled preventive vaccine efficacy trials, especially trials of HIV vaccines. After briefly arguing that certain principal stratification estimands for studying vaccine effects on post-infection outcomes are of genuine scientific interest, the bulk of our commentary argues that the "causal effect predictiveness" (CEP) principal stratification estimand for evaluating immune biomarkers as surrogate endpoints is not of ultimate scientific interest, because it evaluates surrogacy restricted to the setting of a particular vaccine efficacy trial, but is nevertheless useful for guiding the selection of primary immune biomarker endpoints in Phase I/II vaccine trials and for facilitating assessment of transportability/bridging surrogacy.     

A bayesian approach to improved estimation of causal effect predictiveness for a principal surrogate endpoint

Summary The literature on potential outcomes has shown that traditional methods for characterizing surrogate endpoints in clinical trials based only on observed quantities can fail to capture causal relationships between treatments, surrogates, and outcomes. Building on the potential-outcomes formulation of a principal surrogate, we introduce a Bayesian method to estimate the causal effect predictiveness (CEP) surface and quantify a candidate surrogate's utility for reliably predicting clinical outcomes. In considering the full joint distribution of all potentially observable quantities, our Bayesian approach has the following features. First, our approach illuminates implicit assumptions embedded in previously-used estimation strategies that have been shown to result in poor performance. Second, our approach provides tools for making explicit and scientifically-interpretable assumptions regarding associations about which observed data are not informative. Through simulations based on an HIV vaccine trial, we found that the Bayesian approach can produce estimates of the CEP surface with improved performance compared to previous methods. Third, our approach can extend principal-surrogate estimation beyond the previously considered setting of a vaccine trial where the candidate surrogate is constant in one arm of the study. We illustrate this extension through an application to an AIDS therapy trial where the candidate surrogate varies in both treatment arms.     

Statistical Identifiability and the Surrogate Endpoint Problem,with Application to Vaccine Trials

Summary Given a randomized treatment Z, a clinical outcome Y, and a biomarker S measured some fixed time after Z is administered, we may be interested in addressing the surrogate endpoint problem by evaluating whether S can be used to reliably predict the effect of Z on Y. Several recent proposals for the statistical evaluation of surrogate value have been based on the framework of principal stratification. In this article, we consider two principal stratification estimands: joint risks and marginal risks. Joint risks measure causal associations (CAs) of treatment effects on S and Y, providing insight into the surrogate value of the biomarker, but are not statistically identifiable from vaccine trial data. Although marginal risks do not measure CAs of treatment effects, they nevertheless provide guidance for future research, and we describe a data collection scheme and assumptions under which the marginal risks are statistically identifiable. We show how different sets of assumptions affect the identifiability of these estimands; in particular, we depart from previous work by considering the consequences of relaxing the assumption of no individual treatment effects on Y before S is measured. Based on algebraic relationships between joint and marginal risks, we propose a sensitivity analysis approach for assessment of surrogate value, and show that in many cases the surrogate value of a biomarker may be hard to establish, even when the sample size is large.     

Principal stratification in causal inference

Many scientific problems require that treatment comparisons be adjusted for posttreatment variables, but the estimands underlying standard methods are not causal effects. To address this deficiency, we propose a general framework for comparing treatments adjusting for posttreatment variables that yields principal effects based on principal stratification. Principal stratification with respect to a posttreatment variable is a cross-classification of subjects defined by the joint potential values of that posttreatment variable tinder each of the treatments being compared. Principal effects are causal effects within a principal stratum. The key property of principal strata is that they are not affected by treatment assignment and therefore can be used just as any pretreatment covariate. such as age category. As a result, the central property of our principal effects is that they are always causal effects and do not suffer from the complications of standard posttreatment-adjusted estimands. We discuss briefly that such principal causal effects are the link between three recent applications with adjustment for posttreatment variables: (i) treatment noncompliance, (ii) missing outcomes (dropout) following treatment noncompliance. and (iii) censoring by death. We then attack the problem of surrogate or biomarker endpoints, where we show, using principal causal effects, that all current definitions of surrogacy, even when perfectly true, do not generally have the desired interpretation as causal effects of treatment on outcome. We go on to forrmulate estimands based on principal stratification and principal causal effects and show their superiority.     

Predicting treatment effect from surrogate endpoints and historical trials: an extrapolation involving probabilities of a binary outcome or survival to a specific time

Using multiple historical trials with surrogate and true endpoints, we consider various models to predict the effect of treatment on a true endpoint in a target trial in which only a surrogate endpoint is observed. This predicted result is computed using (1) a prediction model (mixture, linear, or principal stratification) estimated from historical trials and the surrogate endpoint of the target trial and (2) a random extrapolation error estimated from successively leaving out each trial among the historical trials. The method applies to either binary outcomes or survival to a particular time that is computed from censored survival data. We compute a 95% confidence interval for the predicted result and validate its coverage using simulation. To summarize the additional uncertainty from using a predicted instead of true result for the estimated treatment effect, we compute its multiplier of standard error. Software is available for download.     

Clinical and biomarker endpoint analysis in neoadjuvant endocrine therapy trials

Neoadjuvant endocrine therapy trials for breast cancer are now a widely accepted investigational approach for oncology cooperative group and pharmaceutical company research programs. However, there remains considerable uncertainty regarding the most suitable endpoints for these studies, in part, because short-term clinical, radiological or biomarker responses have not been fully validated as surrogate endpoints that closely relate to long-term breast cancer outcome. This shortcoming must be addressed before neoadjuvant endocrine treatment can be used as a triage strategy designed to identify patients with endocrine therapy “curable” disease. In this summary, information from published studies is used as a basis to critique clinical trial designs and to suggest experimental endpoints for future validation studies. Three aspects of neoadjuvant endocrine therapy designs are considered: the determination of response; the assessment of surgical outcomes; and biomarker endpoint analysis. Data from the letrozole 024 (LET 024) trial that compared letrozole and tamoxifen is used to illustrate a combined endpoint analysis that integrates both clinical and biomarker information. In addition, the concept of a “cell cycle response” is explored as a simple post-treatment endpoint based on Ki67 analysis that might have properties similar to the pathological complete response endpoint used in neoadjuvant chemotherapy trials.     

Bayesian adaptive trial design for a newly validated surrogate endpoint

The evaluation of surrogate endpoints for primary use in future clinical trials is an increasingly important research area, due to demands for more efficient trials coupled with recent regulatory acceptance of some surrogates as 'valid.' However, little consideration has been given to how a trial that utilizes a newly validated surrogate endpoint as its primary endpoint might be appropriately designed. We propose a novel Bayesian adaptive trial design that allows the new surrogate endpoint to play a dominant role in assessing the effect of an intervention, while remaining realistically cautious about its use. By incorporating multitrial historical information on the validated relationship between the surrogate and clinical endpoints, then subsequently evaluating accumulating data against this relationship as the new trial progresses, we adaptively guard against an erroneous assessment of treatment based upon a truly invalid surrogate. When the joint outcomes in the new trial seem plausible given similar historical trials, we proceed with the surrogate endpoint as the primary endpoint, and do so adaptively-perhaps stopping the trial for early success or inferiority of the experimental treatment, or for futility. Otherwise, we discard the surrogate and switch adaptive determinations to the original primary endpoint. We use simulation to test the operating characteristics of this new design compared to a standard O'Brien-Fleming approach, as well as the ability of our design to discriminate trustworthy from untrustworthy surrogates in hypothetical future trials. Furthermore, we investigate possible benefits using patient-level data from 18 adjuvant therapy trials in colon cancer, where disease-free survival is considered a newly validated surrogate endpoint for overall survival.     

Assessment of DNA flow cytometry as a surrogate end point biomarker in a bladder cancer chemoprevention trial

Although conventional cytology represents the most widely performed cytometric analysis of bladder cancer cells, DNA flow cytometry has, over the past decade, been increasingly used to evaluate cell proliferation and DNA ploidy in cells from bladder washings. We have investigated whether DNA flow cytometry and conventional cytology of epithelial cells obtained from bladder washings provide reliable surrogate endpoint biomarkers in clinical chemoprevention trials. We used cytometric and clinical data from a chemoprevention trial of the synthetic retinoid Fenretinide on 99 patients with superficial bladder cancer. A total of 642 bladder washing specimens obtained from the patients at 4 month intervals was analyzed. Intra-individual agreement and correlation of flow cytometric DNA ploidy (diploid vs. aneuploid), DNA Index, Hyper-Diploid-Fraction (proportion of cells with DNA content higher than 2C), and conventional cytologic examination, as assessed by kappa statistics and Spearman's correlation test, were poor from baseline through 24 months. Moreover, no correlation was found between DNA ploidy and cytology at each time point. The same results were obtained when the analyses were stratified by treatment group. In addition, the association between the results of bladder washing (by either DNA flow cytometry or cytology) and concomitant tumor recurrence was significant only for abnormal cytology, while neither biomarker was predictive of tumor recurrence at the subsequent visit. During the time of this study only four patients progressed to muscle-invasive bladder cancer, indicating the "low-risk" features of the patient population. We conclude that DNA flow cytometry and conventional cytology on epithelial cells obtained from bladder washings do not appear to provide suitable surrogate endpoint biomarkers during the early stages of bladder carcinogenesis.     

The evaluation of multiple surrogate endpoints

Surrogate endpoints are desirable because they typically result in smaller, faster efficacy studies compared with the ones using the clinical endpoints. Research on surrogate endpoints has received substantial attention lately, but most investigations have focused on the validity of using a single biomarker as a surrogate. Our paper studies whether the use of multiple markers can improve inferences about a treatment's effects on a clinical endpoint. We propose a joint model for a time to clinical event and for repeated measures over time on multiple biomarkers that are potential surrogates. This model extends the formulation of Xu and Zeger (2001, in press) and Fawcett and Thomas (1996, Statistics in Medicine 15, 1663-1685). We propose two complementary measures of the relative benefit of multiple surrogates as opposed to a single one. Markov chain Monte Carlo is implemented to estimate model parameters. The methodology is illustrated with an analysis of data from a schizophrenia clinical trial.     

Solutions for surrogacy validation with longitudinal outcomes for a gene therapy

Valid surrogate endpoints S can be used as a substitute for a true outcome of interest T to measure treatment efficacy in a clinical trial. We propose a causal inference approach to validate a surrogate by incorporating longitudinal measurements of the true outcomes using a mixed modeling approach, and we define models and quantities for validation that may vary across the study period using principal surrogacy criteria. We consider a surrogate-dependent treatment efficacy curve that allows us to validate the surrogate at different time points. We extend these methods to accommodate a delayed-start treatment design where all patients eventually receive the treatment. Not all parameters are identified in the general setting. We apply a Bayesian approach for estimation and inference, utilizing more informative prior distributions for selected parameters. We consider the sensitivity of these prior assumptions as well as assumptions of independence among certain counterfactual quantities conditional on pretreatment covariates to improve identifiability. We examine the frequentist properties (bias of point and variance estimates, credible interval coverage) of a Bayesian imputation method. Our work is motivated by a clinical trial of a gene therapy where the functional outcomes are measured repeatedly throughout the trial.     

A simple meta-analytic approach for using a binary surrogate endpoint to predict the effect of intervention on true endpoint

A surrogate endpoint is an endpoint that is obtained sooner, at lower cost, or less invasively than the true endpoint for a health outcome and is used to make conclusions about the effect of intervention on the true endpoint. In this approach, each previous trial with surrogate and true endpoints contributes an estimated predicted effect of intervention on true endpoint in the trial of interest based on the surrogate endpoint in the trial of interest. These predicted quantities are combined in a simple random-effects meta-analysis to estimate the predicted effect of intervention on true endpoint in the trial of interest. Validation involves comparing the average prediction error of the aforementioned approach with (i) the average prediction error of a standard meta-analysis using only true endpoints in the other trials and (ii) the average clinically meaningful difference in true endpoints implicit in the trials. Validation is illustrated using data from multiple randomized trials of patients with advanced colorectal cancer in which the surrogate endpoint was tumor response and the true endpoint was median survival time.     

Properties of a nonparametric test for early comparison of treatments in clinical trials in the presence of surrogate endpoints

A nonparametric test is derived for comparing treatments with respect to the final endpoint in clinical trials in which the final endpoint has been observed for a random subset of patients, but results are available for a surrogate endpoint for a larger sample of patients. The test is an adaptation of the Wilcoxon-Mann-Whitney two-sample test, with an adjustment that involves a comparison of the ranks of the surrogate endpoints between patients with and without final endpoints. The validity of the test depends on the assumption that the patients with final endpoints represent a random sample of the patients registered in the study. This assumption is viable in trials in which the final endpoint is evaluated at a "landmark" timepoint in the patients' natural history. A small sample simulation study demonstrates that the test has a size that is close to the nominal value for all configurations evaluated. When compared with the conventional test based only on the final endpoints, the new test delivers substantial increases in power only when the surrogate endpoint is highly correlated with the true endpoint. Our research indicates that, in the absence of modeling assumptions, auxiliary information derived from surrogate endpoints can provide significant additional information only under special circumstances.     

Assessing treatment effect heterogeneity in clinical trials with blocked binary outcomes

This paper addresses treatment effect heterogeneity (also referred to, more compactly, as 'treatment heterogeneity') in the context of a controlled clinical trial with binary endpoints. Treatment heterogeneity, variation in the true (causal) individual treatment effects, is explored using the concept of the potential outcome. This framework supposes the existance of latent responses for each subject corresponding to each possible treatment. In the context of a binary endpoint, treatment heterogeniety may be represented by the parameter, pi2, the probability that an individual would have a failure on the experimental treatment, if received, and would have a success on control, if received. Previous research derived bounds for pi2 based on matched pairs data. The present research extends this method to the blocked data context. Estimates (and their variances) and confidence intervals for the bounds are derived. We apply the new method to data from a renal disease clinical trial. In this example, bounds based on the blocked data are narrower than the corresponding bounds based only on the marginal success proportions. Some remaining challenges (including the possibility of further reducing bound widths) are discussed.     

On assessing surrogacy in a single trial setting using a semicompeting risks paradigm

Summary .  There has been a recent emphasis on the identification of biomarkers and other biologic measures that may be potentially used as surrogate endpoints in clinical trials. We focus on the setting of data from a single clinical trial. In this article, we consider a framework in which the surrogate must occur before the true endpoint. This suggests viewing the surrogate and true endpoints as semicompeting risks data; this approach is new to the literature on surrogate endpoints and leads to an asymmetrical treatment of the surrogate and true endpoints. However, such a data structure also conceptually complicates many of the previously considered measures of surrogacy in the literature. We propose novel estimation and inferential procedures for the relative effect and adjusted association quantities proposed by Buyse and Molenberghs (1998, Biometrics 54, 1014–1029). The proposed methodology is illustrated with application to simulated data, as well as to data from a leukemia study.     

Overlooked pitfalls in multi-class machine learning classification in radiation oncology and how to avoid them

In radiation oncology, Machine Learning classification publications are typically related to two outcome classes, e.g. the presence or absence of distant metastasis. However, multi-class classification problems also have great clinical relevance, e.g., predicting the grade of a treatment complication following lung irradiation. This work comprised two studies aimed at making work in this domain less prone to statistical blindsides.In multi-class classification, AUC is not defined, whereas correlation coefficients are. It may seem like solely quoting the correlation coefficient value (in lieu of the AUC value) is a suitable choice. In the first study, we illustrated using Monte Carlo (MC) models why this choice is misleading. We also considered the special case where the multiple classes are not ordinal, but nominal, and explained why Pearson or Spearman correlation coefficients are not only providing incomplete information but are actually meaningless.The second study concerned surrogate biomarkers for a clinical endpoint, which have purported benefits including potential for early assessment, being inexpensive, and being non-invasive. Using a MC experiment, we showed how conclusions derived from surrogate markers can be misleading. The simulated endpoint was radiation toxicity (scale of 0–5). The surrogate marker was the true toxicity grade plus a noise term. Five patient cohorts were simulated, including one control. Two of the cohorts were designed to have a statistically significant difference in toxicity. Under 1000 repeated experiments using the biomarker, these two cohorts were often found to be statistically indistinguishable, with the fraction of such occurrences rising with the level of noise.     

Counterfactual links to the proportion of treatment effect explained by a surrogate marker

In a randomized clinical trial, a statistic that measures the proportion of treatment effect on the primary clinical outcome that is explained by the treatment effect on a surrogate outcome is a useful concept. We investigate whether a statistic proposed to estimate this proportion can be given a causal interpretation as defined by models of counterfactual variables. For the situation of binary surrogate and outcome variables, two counterfactual models are considered, both of which include the concept of the proportion of the treatment effect, which acts through the surrogate. In general, the statistic does not equal either of the two proportions from the counterfactual models, and can be substantially different. Conditions are given for which the statistic does equal the counterfactual model proportions. A randomized clinical trial with potential surrogate endpoints is undertaken in a scientific context; this context will naturally place constraints on the parameters of the counterfactual model. We conducted a simulation experiment to investigate what impact these constraints had on the relationship between the proportion explained (PE) statistic and the counterfactual model proportions. We found that observable constraints had very little impact on the agreement between the statistic and the counterfactual model proportions, whereas unobservable constraints could lead to more agreement.     

Biomarker Report from the Phase II Lamotrigine Trial in Secondary Progressive MS – Neurofilament as a Surrogate of Disease Progression

Objective

Lamotrigine trial in SPMS was a randomised control trial to assess whether partial blockade of sodium channels has a neuroprotective effect. The current study was an additional study to investigate the value of neurofilament (NfH) and other biomarkers in predicting prognosis and/or response to treatment.

Methods

SPMS patients who attended the NHNN or the Royal Free Hospital, UK, eligible for inclusion were invited to participate in the biomarker study. Primary outcome was whether lamotrigine would significantly reduce detectable serum NfH at 0-12, 12–24 and 0–24 months compared to placebo. Other serum/plasma and CSF biomarkers were also explored.

Results

Treatment effect by comparing absolute changes in NfH between the lamotrigine and placebo group showed no difference, however based on serum lamotrigine adherence there was significant decline in NfH (NfH 12–24 months p = 0.043, Nfh 0–24 months p = 0.023). Serum NfH correlated with disability: walking times, 9-HPT (non-dominant hand), PASAT, z-score, MSIS-29 (psychological) and EDSS and MRI cerebral atrophy and MTR. Other biomarkers explored in this study were not found to be significantly associated, aside from that of plasma osteopontin.

Conclusions

The relations between NfH and clinical scores of disability and MRI measures of atrophy and disease burden support NfH being a potential surrogate endpoint complementing MRI in neuroprotective trials and sample sizes for such trials are presented here. We did not observe a reduction in NfH levels between the Lamotrigine and placebo arms, however, the reduction in serum NfH levels based on lamotrigine adherence points to a possible neuroprotective effect of lamotrigine on axonal degeneration.     

Validity of Sealant Retention as Surrogate for Caries Prevention – A Systematic Review

Introduction/Aim

To appraise the clinical literature in determining whether loss of complete sealant retention as surrogate endpoint is directly associated with caries occurrence on sealed teeth as its clinical endpoint and to apply the appraised evidence in testing the null-hypothesis that the retention/caries ratio between different types of sealant materials (resin and glass-ionomer cement) is not statistically significant ( = Prentice criterion for surrogate endpoint validity).

Methods

Databases searched PubMed/Medline, Directory of Open Access Journals; IndMed, Scielo. Systematic reviews were checked for suitable trials. The search terms: “fiss* AND seal*” and “fissure AND sealant” were used. Article selection criteria were: clinical trial reporting on the retention and caries occurrence of resin and/or glass-ionomer cement (GIC) fissure sealed permanent molar teeth; minimum 24-month follow-up period; systematic review or meta-analysis. Datasets and information were extracted from accepted trials. The principle outcome measure was the ratio of Risk of loss of complete retention to the Risk of caries occurrence per sealant type (RCR). Risk of bias was assessed in trials and sensitivity analysis with regard to potential confounding factors conducted. The null-hypothesis was tested by graphical and statistical methods.

Results

The risk of loss of complete retention of sealant materials was associated with the risk of caries occurrence for resin but not for GIC based sealants. The difference between RCR values of the two sealant types was statistically significant (p<0.05). The null-hypothesis was rejected.

Conclusions

The current clinical evidence suggests that complete retention of pit and fissure sealants may not be a valid surrogate endpoint for caries prevention as its clinical endpoint. Further research is required to corroborate the current results.     

Causal assessment of surrogacy in a meta-analysis of colorectal cancer trials

When the true end points (T) are difficult or costly to measure, surrogate markers (S) are often collected in clinical trials to help predict the effect of the treatment (Z). There is great interest in understanding the relationship among S, T, and Z. A principal stratification (PS) framework has been proposed by Frangakis and Rubin (2002) to study their causal associations. In this paper, we extend the framework to a multiple trial setting and propose a Bayesian hierarchical PS model to assess surrogacy. We apply the method to data from a large collection of colon cancer trials in which S and T are binary. We obtain the trial-specific causal measures among S, T, and Z, as well as their overall population-level counterparts that are invariant across trials. The method allows for information sharing across trials and reduces the nonidentifiability problem. We examine the frequentist properties of our model estimates and the impact of the monotonicity assumption using simulations. We also illustrate the challenges in evaluating surrogacy in the counterfactual framework that result from nonidentifiability.