A systematic assessment of MHC class II peptide binding predictions and evaluation of a consensus approach

The identification of MHC class II restricted peptide epitopes is an important goal in immunological research. A number of computational tools have been developed for this purpose, but there is a lack of large-scale systematic evaluation of their performance. Herein, we used a comprehensive dataset consisting of more than 10,000 previously unpublished MHC-peptide binding affinities, 29 peptide/MHC crystal structures, and 664 peptides experimentally tested for CD4+ T cell responses to systematically evaluate the performances of publicly available MHC class II binding prediction tools. While in selected instances the best tools were associated with AUC values up to 0.86, in general, class II predictions did not perform as well as historically noted for class I predictions. It appears that the ability of MHC class II molecules to bind variable length peptides, which requires the correct assignment of peptide binding cores, is a critical factor limiting the performance of existing prediction tools. To improve performance, we implemented a consensus prediction approach that combines methods with top performances. We show that this consensus approach achieved best overall performance. Finally, we make the large datasets used publicly available as a benchmark to facilitate further development of MHC class II binding peptide prediction methods.     

NetTepi: an integrated method for the prediction of T cell epitopes

Multiple factors determine the ability of a peptide to elicit a cytotoxic T cell lymphocyte response. Binding to a major histocompatibility complex class I (MHC-I) molecule is one of the most essential factors, as no peptide can become a T cell epitope unless presented on the cell surface in complex with an MHC-I molecule. As such, peptide-MHC (pMHC) binding affinity predictors are currently the premier methods for T cell epitope prediction, and these prediction methods have been shown to have high predictive performances in multiple studies. However, not all MHC-I binders are T cell epitopes, and multiple studies have investigated what additional factors are important for determining the immunogenicity of a peptide. A recent study suggested that pMHC stability plays an important role in determining if a peptide can become a T cell epitope. Likewise, a T cell propensity model has been proposed for identifying MHC binding peptides with amino acid compositions favoring T cell receptor interactions. In this study, we investigate if improved accuracy for T cell epitope discovery can be achieved by integrating predictions for pMHC binding affinity, pMHC stability, and T cell propensity. We show that a weighted sum approach allows pMHC stability and T cell propensity predictions to enrich pMHC binding affinity predictions. The integrated model leads to a consistent and significant increase in predictive performance and we demonstrate how this can be utilized to decrease the experimental workload of epitope screens. The final method, NetTepi, is publically available at www.cbs.dtu.dk/services/NetTepi.     

Quantitative prediction of protein–protein binding affinity with a potential of mean force considering volume correction

Quantitative prediction of protein–protein binding affinity is essential for understanding protein–protein interactions. In this article, an atomic level potential of mean force (PMF) considering volume correction is presented for the prediction of protein–protein binding affinity. The potential is obtained by statistically analyzing X‐ray structures of protein–protein complexes in the Protein Data Bank. This approach circumvents the complicated steps of the volume correction process and is very easy to implement in practice. It can obtain more reasonable pair potential compared with traditional PMF and shows a classic picture of nonbonded atom pair interaction as Lennard‐Jones potential. To evaluate the prediction ability for protein–protein binding affinity, six test sets are examined. Sets 1–5 were used as test set in five published studies, respectively, and set 6 was the union set of sets 1–5, with a total of 86 protein–protein complexes. The correlation coefficient (R) and standard deviation (SD) of fitting predicted affinity to experimental data were calculated to compare the performance of ours with that in literature. Our predictions on sets 1–5 were as good as the best prediction reported in the published studies, and for union set 6, R = 0.76, SD = 2.24 kcal/mol. Furthermore, we found that the volume correction can significantly improve the prediction ability. This approach can also promote the research on docking and protein structure prediction.     

A community resource benchmarking predictions of peptide binding to MHC-I molecules

Recognition of peptides bound to major histocompatibility complex (MHC) class I molecules by T lymphocytes is an essential part of immune surveillance. Each MHC allele has a characteristic peptide binding preference, which can be captured in prediction algorithms, allowing for the rapid scan of entire pathogen proteomes for peptide likely to bind MHC. Here we make public a large set of 48,828 quantitative peptide-binding affinity measurements relating to 48 different mouse, human, macaque, and chimpanzee MHC class I alleles. We use this data to establish a set of benchmark predictions with one neural network method and two matrix-based prediction methods extensively utilized in our groups. In general, the neural network outperforms the matrix-based predictions mainly due to its ability to generalize even on a small amount of data. We also retrieved predictions from tools publicly available on the internet. While differences in the data used to generate these predictions hamper direct comparisons, we do conclude that tools based on combinatorial peptide libraries perform remarkably well. The transparent prediction evaluation on this dataset provides tool developers with a benchmark for comparison of newly developed prediction methods. In addition, to generate and evaluate our own prediction methods, we have established an easily extensible web-based prediction framework that allows automated side-by-side comparisons of prediction methods implemented by experts. This is an advance over the current practice of tool developers having to generate reference predictions themselves, which can lead to underestimating the performance of prediction methods they are not as familiar with as their own. The overall goal of this effort is to provide a transparent prediction evaluation allowing bioinformaticians to identify promising features of prediction methods and providing guidance to immunologists regarding the reliability of prediction tools.     

Structure-based identification of MHC binding peptides: Benchmarking of prediction accuracy

Identification of MHC binding peptides is essential for understanding the molecular mechanism of immune response. However, most of the prediction methods use motifs/profiles derived from experimental peptide binding data for specific MHC alleles, thus limiting their applicability only to those alleles for which such data is available. In this work we have developed a structure-based method which does not require experimental peptide binding data for training. Our method models MHC-peptide complexes using crystal structures of 170 MHC-peptide complexes and evaluates the binding energies using two well known residue based statistical pair potentials, namely Betancourt-Thirumalai (BT) and Miyazawa-Jernigan (MJ) matrices. Extensive benchmarking of prediction accuracy on a data set of 1654 epitopes from class I and class II alleles available in the SYFPEITHI database indicate that BT pair-potential can predict more than 60% of the known binders in case of 14 MHC alleles with AUC values for ROC curves ranging from 0.6 to 0.9. Similar benchmarking on 29,522 class I and class II MHC binding peptides with known IC(50) values in the IEDB database showed AUC values higher than 0.6 for 10 class I alleles and 9 class II alleles in predictions involving classification of a peptide to be binder or non-binder. Comparison with recently available benchmarking studies indicated that, the prediction accuracy of our method for many of the class I and class II MHC alleles was comparable to the sequence based methods, even if it does not use any experimental data for training. It is also encouraging to note that the ranks of true binding peptides could further be improved, when high scoring peptides obtained from pair potential were re-ranked using all atom forcefield and MM/PBSA method.     

Prediction of MHC binding peptide using Gibbs motif sampler,weight matrix and artificial neural network

The identification of MHC restricted epitopes is an important goal in peptide based vaccine and diagnostic development. As wet lab experiments for identification of MHC binding peptide are expensive and time consuming, in silico tools have been developed as fast alternatives, however with low performance. In the present study, we used IEDB training and blind validation datasets for the prediction of peptide binding to fourteen human MHC class I and II molecules using Gibbs motif sampler, weight matrix and artificial neural network methods. As compare to MHC class I predictor based on sequence weighting (Aroc=0.95 and CC=0.56) and artificial neural network (Aroc=0.73 and CC=0.25), MHC class II predictor based on Gibbs sampler did not perform well (Aroc=0.62 and CC=0.19). The predictive accuracy of Gibbs motif sampler in identifying the 9-mer cores of a binding peptide to DRB1 alleles are also limited (40¢), however above the random prediction (14¢). Therefore, the size of dataset (training and validation) and the correct identification of the binding core are the two main factors limiting the performance of MHC class-II binding peptide prediction. Overall, these data suggest that there is substantial room to improve the quality of the core predictions using novel approaches that capture distinct features of MHC-peptide interactions than the current approaches.     

FunFOLDQA: a quality assessment tool for protein-ligand binding site residue predictions

The estimation of prediction quality is important because without quality measures, it is difficult to determine the usefulness of a prediction. Currently, methods for ligand binding site residue predictions are assessed in the function prediction category of the biennial Critical Assessment of Techniques for Protein Structure Prediction (CASP) experiment, utilizing the Matthews Correlation Coefficient (MCC) and Binding-site Distance Test (BDT) metrics. However, the assessment of ligand binding site predictions using such metrics requires the availability of solved structures with bound ligands. Thus, we have developed a ligand binding site quality assessment tool, FunFOLDQA, which utilizes protein feature analysis to predict ligand binding site quality prior to the experimental solution of the protein structures and their ligand interactions. The FunFOLDQA feature scores were combined using: simple linear combinations, multiple linear regression and a neural network. The neural network produced significantly better results for correlations to both the MCC and BDT scores, according to Kendall's τ, Spearman's ρ and Pearson's r correlation coefficients, when tested on both the CASP8 and CASP9 datasets. The neural network also produced the largest Area Under the Curve score (AUC) when Receiver Operator Characteristic (ROC) analysis was undertaken for the CASP8 dataset. Furthermore, the FunFOLDQA algorithm incorporating the neural network, is shown to add value to FunFOLD, when both methods are employed in combination. This results in a statistically significant improvement over all of the best server methods, the FunFOLD method (6.43%), and one of the top manual groups (FN293) tested on the CASP8 dataset. The FunFOLDQA method was also found to be competitive with the top server methods when tested on the CASP9 dataset. To the best of our knowledge, FunFOLDQA is the first attempt to develop a method that can be used to assess ligand binding site prediction quality, in the absence of experimental data.     

Dataset size and composition impact the reliability of performance benchmarks for peptide-MHC binding predictions

Background

It is important to accurately determine the performance of peptide:MHC binding predictions, as this enables users to compare and choose between different prediction methods and provides estimates of the expected error rate. Two common approaches to determine prediction performance are cross-validation, in which all available data are iteratively split into training and testing data, and the use of blind sets generated separately from the data used to construct the predictive method. In the present study, we have compared cross-validated prediction performances generated on our last benchmark dataset from 2009 with prediction performances generated on data subsequently added to the Immune Epitope Database (IEDB) which served as a blind set.

Results

We found that cross-validated performances systematically overestimated performance on the blind set. This was found not to be due to the presence of similar peptides in the cross-validation dataset. Rather, we found that small size and low sequence/affinity diversity of either training or blind datasets were associated with large differences in cross-validated vs. blind prediction performances. We use these findings to derive quantitative rules of how large and diverse datasets need to be to provide generalizable performance estimates.

Conclusion

It has long been known that cross-validated prediction performance estimates often overestimate performance on independently generated blind set data. We here identify and quantify the specific factors contributing to this effect for MHC-I binding predictions. An increasing number of peptides for which MHC binding affinities are measured experimentally have been selected based on binding predictions and thus are less diverse than historic datasets sampling the entire sequence and affinity space, making them more difficult benchmark data sets. This has to be taken into account when comparing performance metrics between different benchmarks, and when deriving error estimates for predictions based on benchmark performance.

Electronic supplementary material

The online version of this article (doi:10.1186/1471-2105-15-241) contains supplementary material, which is available to authorized users.     

Automated generation and evaluation of specific MHC binding predictive tools: ARB matrix applications

Prediction of which peptides can bind major histocompatibility complex (MHC) molecules is commonly used to assist in the identification of T cell epitopes. However, because of the large numbers of different MHC molecules of interest, each associated with different predictive tools, tool generation and evaluation can be a very resource intensive task. A methodology commonly used to predict MHC binding affinity is the matrix or linear coefficients method. Herein, we described Average Relative Binding (ARB) matrix methods that directly predict IC₅₀ values allowing combination of searches involving different peptide sizes and alleles into a single global prediction. A computer program was developed to automate the generation and evaluation of ARB predictive tools. Using an in-house MHC binding database, we generated a total of 85 and 13 MHC class I and class II matrices, respectively. Results from the automated evaluation of tool efficiency are presented. We anticipate that this automation framework will be generally applicable to the generation and evaluation of large numbers of MHC predictive methods and tools, and will be of value to centralize and rationalize the process of evaluation of MHC predictions. MHC binding predictions based on ARB matrices were made available at web server.     

Inferring PDZ domain multi-mutant binding preferences from single-mutant data

Many important cellular protein interactions are mediated by peptide recognition domains. The ability to predict a domain's binding specificity directly from its primary sequence is essential to understanding the complexity of protein-protein interaction networks. One such recognition domain is the PDZ domain, functioning in scaffold proteins that facilitate formation of signaling networks. Predicting the PDZ domain's binding specificity was a part of the DREAM4 Peptide Recognition Domain challenge, the goal of which was to describe, as position weight matrices, the specificity profiles of five multi-mutant ERBB2IP-1 domains. We developed a method that derives multi-mutant binding preferences by generalizing the effects of single point mutations on the wild type domain's binding specificities. Our approach, trained on publicly available ERBB2IP-1 single-mutant phage display data, combined linear regression-based prediction for ligand positions whose specificity is determined by few PDZ positions, and single-mutant position weight matrix averaging for all other ligand columns. The success of our method as the winning entry of the DREAM4 competition, as well as its superior performance over a general PDZ-ligand binding model, demonstrates the advantages of training a model on a well-selected domain-specific data set.     

Limitations of Ab Initio Predictions of Peptide Binding to MHC Class II Molecules

Successful predictions of peptide MHC binding typically require a large set of binding data for the specific MHC molecule that is examined. Structure based prediction methods promise to circumvent this requirement by evaluating the physical contacts a peptide can make with an MHC molecule based on the highly conserved 3D structure of peptide:MHC complexes. While several such methods have been described before, most are not publicly available and have not been independently tested for their performance. We here implemented and evaluated three prediction methods for MHC class II molecules: statistical potentials derived from the analysis of known protein structures; energetic evaluation of different peptide snapshots in a molecular dynamics simulation; and direct analysis of contacts made in known 3D structures of peptide:MHC complexes. These methods are ab initio in that they require structural data of the MHC molecule examined, but no specific peptide:MHC binding data. Moreover, these methods retain the ability to make predictions in a sufficiently short time scale to be useful in a real world application, such as screening a whole proteome for candidate binding peptides. A rigorous evaluation of each methods prediction performance showed that these are significantly better than random, but still substantially lower than the best performing sequence based class II prediction methods available. While the approaches presented here were developed independently, we have chosen to present our results together in order to support the notion that generating structure based predictions of peptide:MHC binding without using binding data is unlikely to give satisfactory results.     

Prediction of MHC class II binding affinity using SMM-align,a novel stabilization matrix alignment method

Background  

Antigen presenting cells (APCs) sample the extra cellular space and present peptides from here to T helper cells, which can be activated if the peptides are of foreign origin. The peptides are presented on the surface of the cells in complex with major histocompatibility class II (MHC II) molecules. Identification of peptides that bind MHC II molecules is thus a key step in rational vaccine design and developing methods for accurate prediction of the peptide:MHC interactions play a central role in epitope discovery. The MHC class II binding groove is open at both ends making the correct alignment of a peptide in the binding groove a crucial part of identifying the core of an MHC class II binding motif. Here, we present a novel stabilization matrix alignment method, SMM-align, that allows for direct prediction of peptide:MHC binding affinities. The predictive performance of the method is validated on a large MHC class II benchmark data set covering 14 HLA-DR (human MHC) and three mouse H2-IA alleles.     

Coupling in silico and in vitro analysis of peptide-MHC binding: a bioinformatic approach enabling prediction of superbinding peptides and anchorless epitopes

The ability to define and manipulate the interaction of peptides with MHC molecules has immense immunological utility, with applications in epitope identification, vaccine design, and immunomodulation. However, the methods currently available for prediction of peptide-MHC binding are far from ideal. We recently described the application of a bioinformatic prediction method based on quantitative structure-affinity relationship methods to peptide-MHC binding. In this study we demonstrate the predictivity and utility of this approach. We determined the binding affinities of a set of 90 nonamer peptides for the MHC class I allele HLA-A*0201 using an in-house, FACS-based, MHC stabilization assay, and from these data we derived an additive quantitative structure-affinity relationship model for peptide interaction with the HLA-A*0201 molecule. Using this model we then designed a series of high affinity HLA-A2-binding peptides. Experimental analysis revealed that all these peptides showed high binding affinities to the HLA-A*0201 molecule, significantly higher than the highest previously recorded. In addition, by the use of systematic substitution at principal anchor positions 2 and 9, we showed that high binding peptides are tolerant to a wide range of nonpreferred amino acids. Our results support a model in which the affinity of peptide binding to MHC is determined by the interactions of amino acids at multiple positions with the MHC molecule and may be enhanced by enthalpic cooperativity between these component interactions.     

Amino acid biophysical properties in the statistical prediction of peptide-MHC class I binding

Background

Peptides derived from endogenous antigens can bind to MHC class I molecules. Those which bind with high affinity can invoke a CD8⁺ immune response, resulting in the destruction of infected cells. Much work in immunoinformatics has involved the algorithmic prediction of peptide binding affinity to various MHC-I alleles. A number of tools for MHC-I binding prediction have been developed, many of which are available on the web.

Results

We hypothesize that peptides predicted by a number of tools are more likely to bind than those predicted by just one tool, and that the likelihood of a particular peptide being a binder is related to the number of tools that predict it, as well as the accuracy of those tools. To this end, we have built and tested a heuristic-based method of making MHC-binding predictions by combining the results from multiple tools. The predictive performance of each individual tool is first ascertained. These performance data are used to derive weights such that the predictions of tools with better accuracy are given greater credence. The combined tool was evaluated using ten-fold cross-validation and was found to signicantly outperform the individual tools when a high specificity threshold is used. It performs comparably well to the best-performing individual tools at lower specificity thresholds. Finally, it also outperforms the combination of the tools resulting from linear discriminant analysis.

Conclusion

A heuristic-based method of combining the results of the individual tools better facilitates the scanning of large proteomes for potential epitopes, yielding more actual high-affinity binders while reporting very few false positives.     

A hydrophobic binding surface on the human immunodeficiency virus type 1 Nef core is critical for association with p21-activated kinase 2

The interaction of human immunodeficiency virus type 1 (HIV-1) Nef with p21-activated kinase 2 (Pak2) has been proposed to play an important role in T-cell activation and disease progression during viral infection. However, the mechanism by which Nef activates Pak2 is poorly understood. Mutations in most Nef motifs previously reported to be required for Pak2 activation (G2, PxxP72, and RR105) also affect other Nef functions, such as CD4 or major histocompatibility complex class I (MHC-I) downregulation. To better understand Nef interactions with Pak2, we performed mutational analysis of three primary HIV-1 Nef clones that exhibited similar capacities for downregulation of CD4 and MHC-I but variable abilities to associate with activated Pak2. Our results demonstrate that Nef amino acids at positions 85, 89, 187, 188, and 191 (L, H, S, R, and F in the clade B consensus, respectively) are critical for Pak2 association. Mutation of these Nef residues dramatically altered association with Pak2 without affecting Nef expression levels or CD4 and MHC-I downregulation. Furthermore, compensation occurred at positions 89 and 191 when both amino acids were substituted. Since residues 85, 89, 187, 188, and 191 cluster on the surface of the Nef core domain in a region distinct from the dimerization and SH3-binding domains, we propose that these Nef residues form part of a unique binding surface specifically involved in association with Pak2. This binding surface includes exposed and recessed hydrophobic residues and may participate in an as-yet-unidentified protein-protein interaction to facilitate Pak2 activation.     

Processing of exogenous antigens for presentation by class I MHC molecules involves post-Golgi peptide exchange influenced by peptide-MHC complex stability and acidic pH

Vacuolar alternate class I MHC (MHC-I) Ag processing allows presentation of exogenous Ag by MHC-I molecules with binding of antigenic peptides to post-Golgi MHC-I molecules. We investigated the role of previously bound peptides and their dissociation in generating peptide-receptive MHC-I molecules. TAP1-knockout macrophages were incubated overnight with an initial exogenous peptide, producing a large cohort of peptide-K(b) complexes that could influence subsequent peptide dissociation/exchange. Initial incubation with FAPGNYPAL, KVVRFDKL, or RGYVYQGL enhanced rather than reduced subsequent binding and presentation of a readout peptide (SIINFEKL or FAPGNYPAL) to T cells. Thus, K(b) molecules may be stabilized by an initial (stabilizing) peptide, enhancing their ability to bind readout peptide and implicating peptide dissociation/exchange. In contrast, incubation with SIINFEKL as stabilizing peptide reduced presentation of readout peptide. SIINFEKL-K(b) complexes were more stable than other peptide-K(b) complexes, which may limit their contribution to peptide exchange. Stabilizing peptides (FAPGNYPAL, KVVRFDKL, or RGYVYQGL) enhanced alternate MHC-I processing of HB101.Crl-OVA (Escherichia coli expressing an OVA fusion protein), indicating that alternate MHC-I Ag processing involves peptide dissociation/exchange. Stabilizing peptide enhanced processing of HB101.Crl-OVA more than presentation of exogenous OVA peptide (SIINFEKL), suggesting that peptide dissociation/exchange may be enhanced in the acidic phagosomal processing environment. Furthermore, exposure of cells to acidic pH increased subsequent binding and presentation of readout peptide. Thus, peptide dissociation/exchange contributes to alternate MHC-I Ag processing and may be influenced by both stability of peptide-MHC-I complexes and pH.     

Quantitative prediction of mouse class I MHC peptide binding affinity using support vector machine regression (SVR) models

Background  

The binding between peptide epitopes and major histocompatibility complex proteins (MHCs) is an important event in the cellular immune response. Accurate prediction of the binding between short peptides and the MHC molecules has long been a principal challenge for immunoinformatics. Recently, the modeling of MHC-peptide binding has come to emphasize quantitative predictions: instead of categorizing peptides as "binders" or "non-binders" or as "strong binders" and "weak binders", recent methods seek to make predictions about precise binding affinities.     

Phage display can select over-hydrophobic sequences that may impair prediction of natural domain-peptide interactions

MOTIVATION: The phage display peptide selection approach is widely used for defining binding specificities of globular domains. PDZ domains recognize partner proteins via C-terminal motifs and are often used as a model for interaction predictions. Here, we investigated to which extent phage display data that were recently published for 54 human PDZ domains can be applied to the prediction of human PDZ-peptide interactions. RESULTS: Promising predictions were obtained for one-third of the 54 PDZ domains. For the other two-thirds, we detected in the phage display peptides an important bias for hydrophobic amino acids that seemed to impair correct predictions. Therefore, phage display-selected peptides may be over-hydrophobic and of high affinity, while natural interaction motifs are rather hydrophilic and mostly combine low affinity with high specificity. We suggest that potential amino acid composition bias should systematically be investigated when applying phage display data to the prediction of specific natural domain-linear motif interactions.