On modeling HIV and T cells in vivo: assessing causal estimators in vaccine trials

The first efficacy trials--named STEP--of a T cell vaccine against HIV/AIDS began in 2004. The unprecedented structure of these trials raised new modeling and statistical challenges. Is it plausible that memory T cells, as opposed to antibodies, can actually prevent infection? If they fail at prevention, to what extent can they ameliorate disease? And how do we estimate efficacy in a vaccine trial with two primary endpoints, one traditional, one entirely novel (viral load after infection), and where the latter may be influenced by selection bias due to the former? In preparation for the STEP trials, biostatisticians developed novel techniques for estimating a causal effect of a vaccine on viral load, while accounting for post-randomization selection bias. But these techniques have not been tested in biologically plausible scenarios. We introduce new stochastic models of T cell and HIV kinetics, making use of new estimates of the rate that cytotoxic T lymphocytes--CTLs; the so-called killer T cells--can kill HIV-infected cells. Based on these models, we make the surprising discovery that it is not entirely implausible that HIV-specific CTLs might prevent infection--as the designers explicitly acknowledged when they chose the endpoints of the STEP trials. By simulating thousands of trials, we demonstrate that the new statistical methods can correctly identify an efficacious vaccine, while protecting against a false conclusion that the vaccine exacerbates disease. In addition to uncovering a surprising immunological scenario, our results illustrate the utility of mechanistic modeling in biostatistics.     

Sensitivity analysis for the assessment of causal vaccine effects on viral load in HIV vaccine trials

Vaccines with limited ability to prevent HIV infection may positively impact the HIV/AIDS pandemic by preventing secondary transmission and disease in vaccine recipients who become infected. To evaluate the impact of vaccination on secondary transmission and disease, efficacy trials assess vaccine effects on HIV viral load and other surrogate endpoints measured after infection. A standard test that compares the distribution of viral load between the infected subgroups of vaccine and placebo recipients does not assess a causal effect of vaccine, because the comparison groups are selected after randomization. To address this problem, we formulate clinically relevant causal estimands using the principal stratification framework developed by Frangakis and Rubin (2002, Biometrics 58, 21-29), and propose a class of logistic selection bias models whose members identify the estimands. Given a selection model in the class, procedures are developed for testing and estimation of the causal effect of vaccination on viral load in the principal stratum of subjects who would be infected regardless of randomization assignment. We show how the procedures can be used for a sensitivity analysis that quantifies how the causal effect of vaccination varies with the presumed magnitude of selection bias.     

Failure time analysis of HIV vaccine effects on viral load and antiretroviral therapy initiation

The world's first efficacy trial of a preventive HIV vaccine was completed in 2003. Study participants who became HIV infected were followed for 2 years and monitored for HIV viral load and initiation of antiretroviral therapy (ART). In order to determine if vaccination may have altered HIV progression in persons who acquired HIV, a pre-specified objective was to compare the time until a composite endpoint between the vaccine and placebo arms, where the composite endpoint is the first event of ART initiation or viral failure (HIV viral load exceeds a threshold x(vl) copies/ml). Specifically, with vaccine efficacy, VE(tau, x(vl)), defined as one minus the ratio (vaccine/placebo) of the cumulative probability of the composite endpoint (with failure threshold x(vl)) occurring by tau months, the aim was to estimate the four parameters {VE(tau, x(vl)): x(vl) is an element of {1500, 10,000, 20,000, 55,000} copies/ml} with simultaneous 95% confidence bands. A Gaussian multipliers simulation method is devised for constructing confidence bands for VE(tau, x(vl)) with x(vl) spanning multiple discrete values or a continuous range. The new method is evaluated in simulations and is applied to the vaccine trial data set.     

The 2-sample problem for failure rates depending on a continuous mark: an application to vaccine efficacy

The efficacy of an HIV vaccine to prevent infection is likely to depend on the genetic variation of the exposing virus. This paper addresses the problem of using data on the HIV sequences that infect vaccine efficacy trial participants to (1) test for vaccine efficacy more powerfully than procedures that ignore the sequence data and (2) evaluate the dependence of vaccine efficacy on the divergence of infecting HIV strains from the HIV strain that is contained in the vaccine. Because hundreds of amino acid sites in each HIV genome are sequenced, it is natural to treat the genetic divergence as a continuous mark variable that accompanies each failure (infection) time. Problems (1) and (2) can then be approached by testing whether the ratio of the mark-specific hazard functions for the vaccine and placebo groups is unity or independent of the mark. We develop nonparametric and semiparametric tests for these null hypotheses and nonparametric techniques for estimating the mark-specific relative risks. The asymptotic properties of the procedures are established. In addition, the methods are studied in simulations and are applied to HIV genetic sequence data collected in the first HIV vaccine efficacy trial.     

Sensitivity analyses comparing outcomes only existing in a subset selected post-randomization, conditional on covariates, with application to HIV vaccine trials

In many experiments, researchers would like to compare between treatments and outcome that only exists in a subset of participants selected after randomization. For example, in preventive HIV vaccine efficacy trials it is of interest to determine whether randomization to vaccine causes lower HIV viral load, a quantity that only exists in participants who acquire HIV. To make a causal comparison and account for potential selection bias we propose a sensitivity analysis following the principal stratification framework set forth by Frangakis and Rubin (2002, Biometrics58, 21-29). Our goal is to assess the average causal effect of treatment assignment on viral load at a given baseline covariate level in the always infected principal stratum (those who would have been infected whether they had been assigned to vaccine or placebo). We assume stable unit treatment values (SUTVA), randomization, and that subjects randomized to the vaccine arm who became infected would also have become infected if randomized to the placebo arm (monotonicity). It is not known which of those subjects infected in the placebo arm are in the always infected principal stratum, but this can be modeled conditional on covariates, the observed viral load, and a specified sensitivity parameter. Under parametric regression models for viral load, we obtain maximum likelihood estimates of the average causal effect conditional on covariates and the sensitivity parameter. We apply our methods to the world's first phase III HIV vaccine trial.     

Resampling-based methods in single and multiple testing for equality of covariance/correlation matrices

Traditional resampling-based tests for homogeneity in covariance matrices across multiple groups resample residuals, that is, data centered by group means. These residuals do not share the same second moments when the null hypothesis is false, which makes them difficult to use in the setting of multiple testing. An alternative approach is to resample standardized residuals, data centered by group sample means and standardized by group sample covariance matrices. This approach, however, has been observed to inflate type I error when sample size is small or data are generated from heavy-tailed distributions. We propose to improve this approach by using robust estimation for the first and second moments. We discuss two statistics: the Bartlett statistic and a statistic based on eigen-decomposition of sample covariance matrices. Both statistics can be expressed in terms of standardized errors under the null hypothesis. These methods are extended to test homogeneity in correlation matrices. Using simulation studies, we demonstrate that the robust resampling approach provides comparable or superior performance, relative to traditional approaches, for single testing and reasonable performance for multiple testing. The proposed methods are applied to data collected in an HIV vaccine trial to investigate possible determinants, including vaccine status, vaccine-induced immune response level and viral genotype, of unusual correlation pattern between HIV viral load and CD4 count in newly infected patients.     

Chop-Lump Tests for Vaccine Trials

Summary This article proposes new tests to compare the vaccine and placebo groups in randomized vaccine trials when a small fraction of volunteers become infected. A simple approach that is consistent with the intent‐to‐treat principle is to assign a score, say W, equal to 0 for the uninfecteds and some postinfection outcome X > 0 for the infecteds. One can then test the equality of this skewed distribution of W between the two groups. This burden of illness (BOI) test was introduced by Chang, Guess, and Heyse (1994, Statistics in Medicine 13 , 1807–1814). If infections are rare, the massive number of 0s in each group tends to dilute the vaccine effect and this test can have poor power, particularly if the X's are not close to zero. Comparing X in just the infecteds is no longer a comparison of randomized groups and can produce misleading conclusions. Gilbert, Bosch, and Hudgens (2003, Biometrics 59 , 531–541) and Hudgens, Hoering, and Self (2003, Statistics in Medicine 22 , 2281–2298) introduced tests of the equality of X in a subgroup—the principal stratum of those “doomed” to be infected under either randomization assignment. This can be more powerful than the BOI approach, but requires unexaminable assumptions. We suggest new “chop‐lump” Wilcoxon and t‐tests (CLW and CLT) that can be more powerful than the BOI tests in certain situations. When the number of volunteers in each group are equal, the chop‐lump tests remove an equal number of zeros from both groups and then perform a test on the remaining W's, which are mostly >0. A permutation approach provides a null distribution. We show that under local alternatives, the CLW test is always more powerful than the usual Wilcoxon test provided the true vaccine and placebo infection rates are the same. We also identify the crucial role of the “gap” between 0 and the X's on power for the t‐tests. The chop‐lump tests are compared to established tests via simulation for planned HIV and malaria vaccine trials. A reanalysis of the first phase III HIV vaccine trial is used to illustrate the method.     

Semiparametric methods for multiple exposure mismeasurement and a bivariate outcome in HIV vaccine trials

Exposure to infection information is important for estimating vaccine efficacy, but it is difficult to collect and prone to missingness and mismeasurement. We discuss study designs that collect detailed exposure information from only a small subset of participants while collecting crude exposure information from all participants and treat estimation of vaccine efficacy in the missing data/measurement error framework. We extend the discordant partner design for HIV vaccine trials of Golm, Halloran, and Longini (1998, Statistics in Medicine, 17, 2335-2352.) to the more complex augmented trial design of Longini, Datta, and Halloran (1996, Journal of Acquired Immune Deficiency Syndromes and Human Retrovirology 13, 440-447) and Datta, Halloran, and Longini (1998, Statistics in Medicine 17, 185-200). The model for this design includes three exposure covariates and both univariate and bivariate outcomes. We adapt recently developed semiparametric missing data methods of Reilly and Pepe (1995, Biometrika 82, 299 314), Carroll and Wand (1991, Journal of the Royal Statistical Society, Series B 53, 573-585), and Pepe and Fleming (1991, Journal of the American Statistical Association 86, 108-113) to the augmented vaccine trial design. We demonstrate with simulated HIV vaccine trial data the improvements in bias and efficiency when combining the different levels of exposure information to estimate vaccine efficacy for reducing both susceptibility and infectiousness. We show that the semiparametric methods estimate both efficacy parameters without bias when the good exposure information is either missing completely at random or missing at random. The pseudolikelihood method of Carroll and Wand (1991) and Pepe and Fleming (1991) was the more efficient of the two semiparametric methods.     

Predicting the impact of a nonsterilizing vaccine against human immunodeficiency virus

Studies of human immunodeficiency virus (HIV) vaccines in animal models suggest that it is difficult to induce complete protection from infection (sterilizing immunity) but that it is possible to reduce the viral load and to slow or prevent disease progression following infection. We have developed an age-structured epidemiological model of the effects of a disease-modifying HIV vaccine that incorporates the intrahost dynamics of infection, a transmission rate and host mortality that depend on the viral load, the possible evolution and transmission of vaccine escape mutant viruses, a finite duration of vaccine protection, and possible changes in sexual behavior. Using this model, we investigated the long-term outcome of a disease-modifying vaccine and utilized uncertainty analysis to quantify the effects of our lack of precise knowledge of various parameters. Our results suggest that the extent of viral load reduction in vaccinated infected individuals (compared to unvaccinated individuals) is the key predictor of vaccine efficacy. Reductions in viral load of about 1 log(10) copies ml(-1) would be sufficient to significantly reduce HIV-associated mortality in the first 20 years after the introduction of vaccination. Changes in sexual risk behavior also had a strong impact on the epidemic outcome. The impact of vaccination is dependent on the population in which it is used, with disease-modifying vaccines predicted to have the most impact in areas of low prevalence and rapid epidemic growth. Surprisingly, the extent to which vaccination alters disease progression, the rate of generation of escape mutants, and the transmission of escape mutants are predicted to have only a weak impact on the epidemic outcome over the first 25 years after the introduction of a vaccine.     

Hypothesis tests for stratified mark‐specific proportional hazards models with missing covariates,with application to HIV vaccine efficacy trials

This article develops hypothesis testing procedures for the stratified mark‐specific proportional hazards model with missing covariates where the baseline functions may vary with strata. The mark‐specific proportional hazards model has been studied to evaluate mark‐specific relative risks where the mark is the genetic distance of an infecting HIV sequence to an HIV sequence represented inside the vaccine. This research is motivated by analyzing the RV144 phase 3 HIV vaccine efficacy trial, to understand associations of immune response biomarkers on the mark‐specific hazard of HIV infection, where the biomarkers are sampled via a two‐phase sampling nested case‐control design. We test whether the mark‐specific relative risks are unity and how they change with the mark. The developed procedures enable assessment of whether risk of HIV infection with HIV variants close or far from the vaccine sequence are modified by immune responses induced by the HIV vaccine; this question is interesting because vaccine protection occurs through immune responses directed at specific HIV sequences. The test statistics are constructed based on augmented inverse probability weighted complete‐case estimators. The asymptotic properties and finite‐sample performances of the testing procedures are investigated, demonstrating double‐robustness and effectiveness of the predictive auxiliaries to recover efficiency. The finite‐sample performance of the proposed tests are examined through a comprehensive simulation study. The methods are applied to the RV144 trial.     

Human immunodeficiency virus (HIV) vaccine trials: a novel assay for differential diagnosis of HIV infections in the face of vaccine-generated antibodies

All current human immunodeficiency virus (HIV) vaccine candidates contain multiple viral components and elicit antibodies that react positively in licensed HIV diagnostic tests, which contain similar viral products. Thus, vaccine trial participants could be falsely diagnosed as infected with HIV. Additionally, uninfected, seropositive vaccinees may encounter long-term social and economic harms. Moreover, this also interferes with early detection of true HIV infections during preventive HIV vaccine trials. An HIV-seropositive test result among uninfected vaccine trial participants is a major public health concern for volunteers who want to participate in future HIV vaccine trials. Based on the increased number of HIV vaccines being tested globally, it is essential to differentiate vaccine- from virus-induced antibodies. Using a whole-HIV-genome phage display library, we identified conserved sequences in Env-gp41 and Gag-p6 which are recognized soon after infection, do not contain protective epitopes, and are not part of most current HIV vaccines. We established a new HIV serodetection assay based on these peptides. To date, this assay, termed HIV-SELECTEST, demonstrates >99% specificity and sensitivity. Importantly, in testing of plasma samples from multiple HIV vaccine trials, uninfected trial participants scored negative, while all intercurrent infections were detected within 1 to 3 months of HIV infection. The new HIV-SELECTEST is a simple but robust diagnostic tool for easy implementation in HIV vaccine trials and blood banks worldwide.     

Nonhuman primate models and the failure of the Merck HIV-1 vaccine in humans

The adenovirus type 5 (Ad5)-based vaccine developed by Merck failed to either prevent HIV-1 infection or suppress viral load in subsequently infected subjects in the STEP human Phase 2b efficacy trial. Analogous vaccines had previously also failed in the simian immunodeficiency virus (SIV) challenge-rhesus macaque model. In contrast, vaccine protection studies that used challenge with a chimeric simian-human immunodeficiency virus (SHIV89.6P) in macaques did not predict the human trial results. Ad5 vector-based vaccines did not protect macaques from infection after SHIV89.6P challenge but did cause a substantial reduction in viral load and a preservation of CD4+ T cell counts after infection, findings that were not reproduced in the human trials. Although the SIV challenge model is incompletely validated, we propose that its expanded use can help facilitate the prioritization of candidate HIV-1 vaccines, ensuring that resources are focused on the most promising candidates. Vaccine designers must now develop T cell vaccine strategies that reduce viral load after heterologous challenge.     

Potential Future Impact of a Partially Effective HIV Vaccine in a Southern African Setting

Background

It is important for public health and within the HIV vaccine development field to understand the potential population level impact of an HIV vaccine of partial efficacy—both in preventing infection and in reducing viral load in vaccinated individuals who become infected—in the context of a realistic future implementation scenario in resource limited settings.

Methods

An individual level model of HIV transmission, progression and the effect of antiretroviral therapy was used to predict the outcome to 2060 of introduction in 2025 of a partially effective vaccine with various combinations of efficacy characteristics, in the context of continued ART roll-out in southern Africa.

Results

In the context of our base case epidemic (in 2015 HIV prevalence 28% and incidence 1.7 per 100 person years), a vaccine with only 30% preventative efficacy could make a substantial difference in the rate with which HIV incidence declines; the impact on incidence in relative terms is projected to increase over time, with a projected 67% lower HIV incidence in 2060 compared with no vaccine introduction. The projected mean decline in the general adult population death rate 2040–2060 is 11%. A vaccine with no prevention efficacy but which reduces viral load by 1 log is predicted to result in a modest (14%) reduction in HIV incidence and an 8% reduction in death rate in the general adult population (mean 2040–2060). These effects were broadly similar in multivariable uncertainty analysis.

Interpretation

Introduction of a partially effective preventive HIV vaccine would make a substantial long-term impact on HIV epidemics in southern Africa, in addition to the effects of ART. Development of an HIV vaccine, even of relatively low apparent efficacy at the individual level, remains a critical global public health goal.     

Assessing Vaccine Effects in Repeated Low‐Dose Challenge Experiments

Summary Evaluation of HIV vaccine candidates in nonhuman primates (NHPs) is a critical step toward developing a successful vaccine to control the HIV pandemic. Historically, HIV vaccine regimens have been tested in NHPs by administering a single high dose of the challenge virus. More recently, evaluation of candidate HIV vaccines has entailed repeated low‐dose challenges, which more closely mimic typical exposure in natural transmission settings. In this article, we consider evaluation of the type and magnitude of vaccine efficacy from such experiments. Based on the principal stratification framework, we also address evaluation of potential immunological surrogate endpoints for infection.     

Combining probability from independent tests: the weighted Z-method is superior to Fisher's approach

The most commonly used method in evolutionary biology for combining information across multiple tests of the same null hypothesis is Fisher's combined probability test. This note shows that an alternative method called the weighted Z-test has more power and more precision than does Fisher's test. Furthermore, in contrast to some statements in the literature, the weighted Z-method is superior to the unweighted Z-transform approach. The results in this note show that, when combining P-values from multiple tests of the same hypothesis, the weighted Z-method should be preferred.     

Commentary on "Principal stratification - a goal or a tool?" by Judea Pearl

This commentary takes up Pearl's welcome challenge to clearly articulate the scientific value of principal stratification estimands that we and colleagues have investigated, in the area of randomized placebo-controlled preventive vaccine efficacy trials, especially trials of HIV vaccines. After briefly arguing that certain principal stratification estimands for studying vaccine effects on post-infection outcomes are of genuine scientific interest, the bulk of our commentary argues that the "causal effect predictiveness" (CEP) principal stratification estimand for evaluating immune biomarkers as surrogate endpoints is not of ultimate scientific interest, because it evaluates surrogacy restricted to the setting of a particular vaccine efficacy trial, but is nevertheless useful for guiding the selection of primary immune biomarker endpoints in Phase I/II vaccine trials and for facilitating assessment of transportability/bridging surrogacy.     

A cautionary note on exact unconditional inference for a difference between two independent binomial proportions

Fisher's exact test for comparing response proportions in a randomized experiment can be overly conservative when the group sizes are small or when the response proportions are close to zero or one. This is primarily because the null distribution of the test statistic becomes too discrete, a partial consequence of the inference being conditional on the total number of responders. Accordingly, exact unconditional procedures have gained in popularity, on the premise that power will increase because the null distribution of the test statistic will presumably be less discrete. However, we caution researchers that a poor choice of test statistic for exact unconditional inference can actually result in a substantially less powerful analysis than Fisher's conditional test. To illustrate, we study a real example and provide exact test size and power results for several competing tests, for both balanced and unbalanced designs. Our results reveal that Fisher's test generally outperforms exact unconditional tests based on using as the test statistic either the observed difference in proportions, or the observed difference divided by its estimated standard error under the alternative hypothesis, the latter for unbalanced designs only. On the other hand, the exact unconditional test based on the observed difference divided by its estimated standard error under the null hypothesis (score statistic) outperforms Fisher's test, and is recommended. Boschloo's test, in which the p-value from Fisher's test is used as the test statistic in an exact unconditional test, is uniformly more powerful than Fisher's test, and is also recommended.     

Genome scanning tests for comparing amino acid sequences between groups

Summary .   Consider a placebo-controlled preventive HIV vaccine efficacy trial. An HIV amino acid sequence is measured from each volunteer who acquires HIV, and these sequences are aligned together with the reference HIV sequence represented in the vaccine. We develop genome scanning methods to identify positions at which the amino acids in infected vaccine recipient sequences either (A) are more divergent from the reference amino acid than the amino acids in infected placebo recipient sequences or (B) have a different frequency distribution than the placebo sequences, irrespective of a reference amino acid. We consider t -test-type statistics for problem A and Euclidean, Mahalanobis, and Kullback–Leibler-type statistics for problem B. The test statistics incorporate weights to reflect biological information contained in different amino acid positions and mismatches. Position-specific p -values are obtained by approximating the null distribution of the statistics either by a permutation procedure or by a nonparametric estimation. A permutation method is used to estimate a cut-off p -value to control the per comparison error rate at a prespecified level. The methods are examined in simulations and are applied to two HIV examples. The methods for problem B address the general problem of comparing discrete frequency distributions between groups in a high-dimensional data setting.     

Estimation of Stratified Mark-Specific Proportional Hazards Models Under Two-Phase Sampling with Application to HIV Vaccine Efficacy Trials

An objective of preventive HIV vaccine efficacy trials is to understand how vaccine-induced immune responses to specific protein sequences of HIV-1 associate with subsequent infection with specific sequences of HIV, where the immune response biomarkers are measured in vaccine recipients via a two-phase sampling design. Motivated by this objective, we investigate the stratified mark-specific proportional hazards model under two-phase biomarker sampling, where the mark is the genetic distance of an infecting HIV-1 sequence to an HIV-1 sequence represented inside the vaccine. Estimation and inference procedures based on inverse probability weighting of complete-cases and on augmented inverse probability weighting of complete-cases are developed. Asymptotic properties of the estimators are derived and their finite-sample performances are examined in simulation studies. The methods are shown to have satisfactory performance and are applied to the RV144 vaccine trial to assess whether immune response correlates of HIV-1 infection are stronger for HIV-1 infecting sequences similar to the vaccine than for sequences distant from the vaccine.     

Estimation of Population Vaccination Effectiveness from HIV Vaccine Trials

Longini , Datta , and Halloran (1996) proposed to design HIV vaccine trials in a way that will permit the simultaneous estimation of the vaccine effects on susceptibility to infection and on infectiousness of vaccine brak-throughs. The main feature of their design is the inclusion of steady partners of trial participants. They estimate four parameters from the vaccine trial: the probability that a susceptible person will become infected from his/her steady partner, the probability of becoming infected from outside the partnership, the vaccine efficacy for susceptibility and the vaccine efficacy for infectiousness. We show how the estimates of these parameters can be used to predict the attack rate in a given population during a specified period following mass-vaccination. This is an iterative method, as the attack rate depends on the HIV prevalence which, in turn, depends on the number of new cases during that period. The same method is also used to estimate the attack rate in that population during the same period in the absence of vaccination. The estimated attack rates allow us to estimate the population vaccination effectiveness, defined as the fraction HIV cases prevented by a vaccination program.