Score_set: A CAPRI benchmark for scoring protein complexes

Critical Assessment of PRedicted Interactions (CAPRI) has proven to be a catalyst for the development of docking algorithms. An essential step in docking is the scoring of predicted binding modes in order to identify stable complexes. In 2005, CAPRI introduced the scoring experiment, where upon completion of a prediction round, a larger set of models predicted by different groups and comprising both correct and incorrect binding modes, is made available to all participants for testing new scoring functions independently from docking calculations. Here we present an expanded benchmark data set for testing scoring functions, which comprises the consolidated ensemble of predicted complexes made available in the CAPRI scoring experiment since its inception. This consolidated scoring benchmark contains predicted complexes for 15 published CAPRI targets. These targets were subjected to 23 CAPRI assessments, due to existence of multiple binding modes for some targets. The benchmark contains more than 19,000 protein complexes. About 10% of the complexes represent docking predictions of acceptable quality or better, the remainder represent incorrect solutions (decoys). The benchmark set contains models predicted by 47 different predictor groups including web servers, which use different docking and scoring procedures, and is arguably as diverse as one may expect, representing the state of the art in protein docking. The data set is publicly available at the following URL: http://cb.iri.univ‐lille1.fr/Users/lensink/Score_set . Proteins 2014; 82:3163–3169. © 2014 Wiley Periodicals, Inc.     

Large oligomeric complex structures can be computationally assembled by efficiently combining docked interfaces

Macromolecular oligomeric assemblies are involved in many biochemical processes of living organisms. The benefits of such assemblies in crowded cellular environments include increased reaction rates, efficient feedback regulation, cooperativity and protective functions. However, an atom‐level structural determination of large assemblies is challenging due to the size of the complex and the difference in binding affinities of the involved proteins. In this study, we propose a novel combinatorial greedy algorithm for assembling large oligomeric complexes from information on the approximate position of interaction interfaces of pairs of monomers in the complex. Prior information on complex symmetry is not required but rather the symmetry is inferred during assembly. We implement an efficient geometric score, the transformation match score, that bypasses the model ranking problems of state‐of‐the‐art scoring functions by scoring the similarity between the inferred dimers of the same monomer simultaneously with different binding partners in a (sub)complex with a set of pregenerated docking poses. We compiled a diverse benchmark set of 308 homo and heteromeric complexes containing 6 to 60 monomers. To explore the applicability of the method, we considered 48 sets of parameters and selected those three sets of parameters, for which the algorithm can correctly reconstruct the maximum number, namely 252 complexes (81.8%) in, at least one of the respective three runs. The crossvalidation coverage, that is, the mean fraction of correctly reconstructed benchmark complexes during crossvalidation, was 78.1%, which demonstrates the ability of the presented method to correctly reconstruct topology of a large variety of biological complexes. Proteins 2015; 83:1887–1899. © 2015 The Authors. Proteins: Structure, Function, and Bioinformatics Published by Wiley Periodicals, Inc.     

How different from random are docking predictions when ranked by scoring functions?

Docking algorithms predict the structure of protein–protein interactions. They sample the orientation of two unbound proteins to produce various predictions about their interactions, followed by a scoring step to rank the predictions. We present a statistical assessment of scoring functions used to rank near‐native orientations, applying our statistical analysis to a benchmark dataset of decoys of protein–protein complexes and assessing the statistical significance of the outcome in the Critical Assessment of PRedicted Interactions (CAPRI) scoring experiment. A P value was assigned that depended on the number of near‐native structures in the sampling. We studied the effect of filtering out redundant structures and tested the use of pair‐potentials derived using ZDock and ZRank. Our results show that for many targets, it is not possible to determine when a successful reranking performed by scoring functions results merely from random choice. This analysis reveals that changes should be made in the design of the CAPRI scoring experiment. We propose including the statistical assessment in this experiment either at the preprocessing or the evaluation step. Proteins 2010. © 2010 Wiley‐Liss, Inc.     

Integrative modeling of protein-protein interactions with pyDock for the new docking challenges

The seventh CAPRI edition imposed new challenges to the modeling of protein-protein complexes, such as multimeric oligomerization, protein-peptide, and protein-oligosaccharide interactions. Many of the proposed targets needed the efficient integration of rigid-body docking, template-based modeling, flexible optimization, multiparametric scoring, and experimental restraints. This was especially relevant for the multimolecular assemblies proposed in the CASP12-CAPRI37 and CASP13-CAPRI46 joint rounds, which were described and evaluated elsewhere. Focusing on the purely CAPRI targets of this edition (rounds 38-45), we have participated in all 17 assessed targets (considering heteromeric and homomeric interfaces in T125 as two separate targets) both as predictors and as scorers, by using integrative modeling based on our docking and scoring approaches: pyDock, IRaPPA, and LightDock. In the protein-protein and protein-peptide targets, we have also participated with our webserver (pyDockWeb). On these 17 CAPRI targets, we submitted acceptable models (or better) within our top 10 models for 10 targets as predictors, 13 targets as scorers, and 4 targets as servers. In summary, our participation in this CAPRI edition confirmed the capabilities of pyDock for the scoring of docking models, increasingly used within the context of integrative modeling of protein interactions and multimeric assemblies.     

Discriminating physiological from non-physiological interfaces in structures of protein complexes: A community-wide study

Reliably scoring and ranking candidate models of protein complexes and assigning their oligomeric state from the structure of the crystal lattice represent outstanding challenges. A community-wide effort was launched to tackle these challenges. The latest resources on protein complexes and interfaces were exploited to derive a benchmark dataset consisting of 1677 homodimer protein crystal structures, including a balanced mix of physiological and non-physiological complexes. The non-physiological complexes in the benchmark were selected to bury a similar or larger interface area than their physiological counterparts, making it more difficult for scoring functions to differentiate between them. Next, 252 functions for scoring protein-protein interfaces previously developed by 13 groups were collected and evaluated for their ability to discriminate between physiological and non-physiological complexes. A simple consensus score generated using the best performing score of each of the 13 groups, and a cross-validated Random Forest (RF) classifier were created. Both approaches showed excellent performance, with an area under the Receiver Operating Characteristic (ROC) curve of 0.93 and 0.94, respectively, outperforming individual scores developed by different groups. Additionally, AlphaFold2 engines recalled the physiological dimers with significantly higher accuracy than the non-physiological set, lending support to the reliability of our benchmark dataset annotations. Optimizing the combined power of interface scoring functions and evaluating it on challenging benchmark datasets appears to be a promising strategy.     

Energy-based graph convolutional networks for scoring protein docking models

Structural information about protein-protein interactions, often missing at the interactome scale, is important for mechanistic understanding of cells and rational discovery of therapeutics. Protein docking provides a computational alternative for such information. However, ranking near-native docked models high among a large number of candidates, often known as the scoring problem, remains a critical challenge. Moreover, estimating model quality, also known as the quality assessment problem, is rarely addressed in protein docking. In this study, the two challenging problems in protein docking are regarded as relative and absolute scoring, respectively, and addressed in one physics-inspired deep learning framework. We represent protein and complex structures as intra- and inter-molecular residue contact graphs with atom-resolution node and edge features. And we propose a novel graph convolutional kernel that aggregates interacting nodes’ features through edges so that generalized interaction energies can be learned directly from 3D data. The resulting energy-based graph convolutional networks (EGCN) with multihead attention are trained to predict intra- and inter-molecular energies, binding affinities, and quality measures (interface RMSD) for encounter complexes. Compared to a state-of-the-art scoring function for model ranking, EGCN significantly improves ranking for a critical assessment of predicted interactions (CAPRI) test set involving homology docking; and is comparable or slightly better for Score_set, a CAPRI benchmark set generated by diverse community-wide docking protocols not known to training data. For Score_set quality assessment, EGCN shows about 27% improvement to our previous efforts. Directly learning from 3D structure data in graph representation, EGCN represents the first successful development of graph convolutional networks for protein docking.     

Scoring protein interaction decoys using exposed residues (SPIDER): a novel multibody interaction scoring function based on frequent geometric patterns of interfacial residues

Accurate prediction of the structure of protein-protein complexes in computational docking experiments remains a formidable challenge. It has been recognized that identifying native or native-like poses among multiple decoys is the major bottleneck of the current scoring functions used in docking. We have developed a novel multibody pose-scoring function that has no theoretical limit on the number of residues contributing to the individual interaction terms. We use a coarse-grain representation of a protein-protein complex where each residue is represented by its side chain centroid. We apply a computational geometry approach called Almost-Delaunay tessellation that transforms protein-protein complexes into a residue contact network, or an undirectional graph where vertex-residues are nodes connected by edges. This treatment forms a family of interfacial graphs representing a dataset of protein-protein complexes. We then employ frequent subgraph mining approach to identify common interfacial residue patterns that appear in at least a subset of native protein-protein interfaces. The geometrical parameters and frequency of occurrence of each "native" pattern in the training set are used to develop the new SPIDER scoring function. SPIDER was validated using standard "ZDOCK" benchmark dataset that was not used in the development of SPIDER. We demonstrate that SPIDER scoring function ranks native and native-like poses above geometrical decoys and that it exceeds in performance a popular ZRANK scoring function. SPIDER was ranked among the top scoring functions in a recent round of CAPRI (Critical Assessment of PRedicted Interactions) blind test of protein-protein docking methods.     

Novel sampling strategies and a coarse-grained score function for docking homomers,flexible heteromers,and oligosaccharides using Rosetta in CAPRI rounds 37–45

Critical Assessment of PRediction of Interactions (CAPRI) rounds 37 through 45 introduced larger complexes, new macromolecules, and multistage assemblies. For these rounds, we used and expanded docking methods in Rosetta to model 23 target complexes. We successfully predicted 14 target complexes and recognized and refined near-native models generated by other groups for two further targets. Notably, for targets T110 and T136, we achieved the closest prediction of any CAPRI participant. We created several innovative approaches during these rounds. Since round 39 (target 122), we have used the new RosettaDock 4.0, which has a revamped coarse-grained energy function and the ability to perform conformer selection during docking with hundreds of pregenerated protein backbones. Ten of the complexes had some degree of symmetry in their interactions, so we tested Rosetta SymDock, realized its shortcomings, and developed the next-generation symmetric docking protocol, SymDock2, which includes docking of multiple backbones and induced-fit refinement. Since the last CAPRI assessment, we also developed methods for modeling and designing carbohydrates in Rosetta, and we used them to successfully model oligosaccharide-protein complexes in round 41. Although the results were broadly encouraging, they also highlighted the pressing need to invest in (a) flexible docking algorithms with the ability to model loop and linker motions and in (b) new sampling and scoring methods for oligosaccharide-protein interactions.     

A combination of rescoring and refinement significantly improves protein docking performance

To determine the structures of protein-protein interactions, protein docking is a valuable tool that complements experimental methods to characterize protein complexes. Although protein docking can often produce a near-native solution within a set of global docking predictions, there are sometimes predictions that require refinement to elucidate correct contacts and conformation. Previously, we developed the ZRANK algorithm to rerank initial docking predictions from ZDOCK, a docking program developed by our lab. In this study, we have applied the ZRANK algorithm toward refinement of protein docking models in conjunction with the protein docking program RosettaDock. This was performed by reranking global docking predictions from ZDOCK, performing local side chain and rigid-body refinement using RosettaDock, and selecting the refined model based on ZRANK score. For comparison, we examined using RosettaDock score instead of ZRANK score, and a larger perturbation size for the RosettaDock search, and determined that the larger RosettaDock perturbation size with ZRANK scoring was optimal. This method was validated on a protein-protein docking benchmark. For refining docking benchmark predictions from the newest ZDOCK version, this led to improved structures of top-ranked hits in 20 of 27 cases, and an increase from 23 to 27 cases with hits in the top 20 predictions. Finally, we optimized the ZRANK energy function using refined models, which provides a significant improvement over the original ZRANK energy function. Using this optimized function and the refinement protocol, the numbers of cases with hits ranked at number one increased from 12 to 19 and from 7 to 15 for two different ZDOCK versions. This shows the effective combination of independently developed docking protocols (ZDOCK/ZRANK, and RosettaDock), indicating that using diverse search and scoring functions can improve protein docking results.     

Ranking multiple docking solutions based on the conservation of inter‐residue contacts

Molecular docking is the method of choice for investigating the molecular basis of recognition in a large number of functional protein complexes. However, correctly scoring the obtained docking solutions (decoys) to rank native‐like (NL) conformations in the top positions is still an open problem. Herein we present CONSRANK, a simple and effective tool to rank multiple docking solutions, which relies on the conservation of inter‐residue contacts in the analyzed decoys ensemble. First it calculates a conservation rate for each inter‐residue contact, then it ranks decoys according to their ability to match the more frequently observed contacts. We applied CONSRANK to 102 targets from three different benchmarks, RosettaDock, DOCKGROUND, and Critical Assessment of PRedicted Interactions (CAPRI). The method performs consistently well, both in terms of NL solutions ranked in the top positions and of values of the area under the receiver operating characteristic curve. Its ideal application is to solutions coming from different docking programs and procedures, as in the case of CAPRI targets. For all the analyzed CAPRI targets where a comparison is feasible, CONSRANK outperforms the CAPRI scorers. The fraction of NL solutions in the top ten positions in the RosettaDock, DOCKGROUND, and CAPRI benchmarks is enriched on average by a factor of 3.0, 1.9, and 9.9, respectively. Interestingly, CONSRANK is also able to specifically single out the high/medium quality (HMQ) solutions from the docking decoys ensemble: it ranks 46.2 and 70.8% of the total HMQ solutions available for the RosettaDock and CAPRI targets, respectively, within the top 20 positions. Proteins 2013. © 2013 Wiley Periodicals, Inc.     

A combinatorial score to distinguish biological and nonbiological protein-protein interfaces

With the large amount of protein-protein complex structural data available, to understand the key features governing the specificity of protein-protein recognition and to define a suitable scoring function for protein-protein interaction predictions, we have analyzed the protein interfaces from geometric and energetic points of view. Atom-based potential of mean force (PMFScore), packing density, contact size, and geometric complementarity are calculated for crystal contacts in 74 homodimers and 91 monomers, which include real biological interactions in dimers and nonbiological contacts in monomers and dimers. Simple cutoffs were developed for single and combinatorial parameters to distinguish biological and nonbiological contacts. The results show that PMFScore is a better discriminator between biological and nonbiological interfaces comparable in size. The combination of PMFScore and contact size is the most powerful pairwise discriminator. A combinatorial score (CFPScore) based on the four parameters was developed, which gives the success rate of the homodimer discrimination of 96.6% and error rate of the monomer discrimination of 6.0% and 19.8% according to Valdar's and our definition, respectively. Compared with other statistical learning models, the cutoffs for the four parameters and their combinations are directly based on physical models, simple, and can be easily applied to protein-protein interface analysis and docking studies.     

Native fold and docking pose discrimination by the same residue‐based scoring function

Structure prediction and quality assessment are crucial steps in modeling native protein conformations. Statistical potentials are widely used in related algorithms, with different parametrizations typically developed for different contexts such as folding protein monomers or docking protein complexes. Here, we describe BACH‐SixthSense, a single residue‐based statistical potential that can be successfully employed in both contexts. BACH‐SixthSense shares the same approach as BACH, a knowledge‐based potential originally developed to score monomeric protein structures. A term that penalizes steric clashes as well as the distinction between polar and apolar sidechain‐sidechain contacts are crucial novel features of BACH‐SixthSense. The performance of BACH‐SixthSense in discriminating correctly the native structure among a competing set of decoys is significantly higher than other state‐of‐the‐art scoring functions, that were specifically trained for a single context, for both monomeric proteins (QMEAN, Rosetta, RF_CB_SRS_OD, benchmarked on CASP targets) and protein dimers (IRAD, Rosetta, PIE*PISA, HADDOCK, FireDock, benchmarked on 14 CAPRI targets). The performance of BACH‐SixthSense in recognizing near‐native docking poses within CAPRI decoy sets is good as well. Proteins 2015; 83:621–630. © 2015 Wiley Periodicals, Inc.     

Structural templates for comparative protein docking

Structural characterization of protein‐protein interactions is important for understanding life processes. Because of the inherent limitations of experimental techniques, such characterization requires computational approaches. Along with the traditional protein‐protein docking (free search for a match between two proteins), comparative (template‐based) modeling of protein‐protein complexes has been gaining popularity. Its development puts an emphasis on full and partial structural similarity between the target protein monomers and the protein‐protein complexes previously determined by experimental techniques (templates). The template‐based docking relies on the quality and diversity of the template set. We present a carefully curated, nonredundant library of templates containing 4950 full structures of binary complexes and 5936 protein‐protein interfaces extracted from the full structures at 12 Å distance cut‐off. Redundancy in the libraries was removed by clustering the PDB structures based on structural similarity. The value of the clustering threshold was determined from the analysis of the clusters and the docking performance on a benchmark set. High structural quality of the interfaces in the template and validation sets was achieved by automated procedures and manual curation. The library is included in the Dockground resource for molecular recognition studies at http://dockground.bioinformatics.ku.edu . Proteins 2015; 83:1563–1570. © 2014 Wiley Periodicals, Inc.     

Gene ontology improves template selection in comparative protein docking

Structural characterization of protein-protein interactions is essential for our ability to study life processes at the molecular level. Computational modeling of protein complexes (protein docking) is important as the source of their structure and as a way to understand the principles of protein interaction. Rapidly evolving comparative docking approaches utilize target/template similarity metrics, which are often based on the protein structure. Although the structural similarity, generally, yields good performance, other characteristics of the interacting proteins (eg, function, biological process, and localization) may improve the prediction quality, especially in the case of weak target/template structural similarity. For the ranking of a pool of models for each target, we tested scoring functions that quantify similarity of Gene Ontology (GO) terms assigned to target and template proteins in three ontology domains—biological process, molecular function, and cellular component (GO-score). The scoring functions were tested in docking of bound, unbound, and modeled proteins. The results indicate that the combined structural and GO-terms functions improve the scoring, especially in the twilight zone of structural similarity, typical for protein models of limited accuracy.     

Template-based modeling of diverse protein interactions in CAPRI rounds 38-45

Structures of proteins complexed with other proteins, peptides, or ligands are essential for investigation of molecular mechanisms. However, the experimental structures of protein complexes of interest are often not available. Therefore, computational methods are widely used to predict these structures, and, of those methods, template-based modeling is the most successful. In the rounds 38-45 of the Critical Assessment of PRediction of Interactions (CAPRI), we applied template-based modeling for 9 of 11 protein-protein and protein-peptide interaction targets, resulting in medium and high-quality models for six targets. For the protein-oligosaccharide docking targets, we used constraints derived from template structures, and generated models of at least acceptable quality for most of the targets. Apparently, high flexibility of oligosaccharide molecules was the main cause preventing us from obtaining models of higher quality. We also participated in the CAPRI scoring challenge, the goal of which was to identify the highest quality models from a large pool of decoys. In this experiment, we tested VoroMQA, a scoring method based on interatomic contact areas. The results showed VoroMQA to be quite effective in scoring strongly binding and obligatory protein complexes, but less successful in the case of transient interactions. We extensively used manual intervention in both CAPRI modeling and scoring experiments. This oftentimes allowed us to select the correct templates from available alternatives and to limit the search space during the model scoring.     

Scoring a diverse set of high-quality docked conformations: a metascore based on electrostatic and desolvation interactions

Predicting protein-protein interactions involves sampling and scoring docked conformations. Barring some large structural rearrangement, rapidly sampling the space of docked conformations is now a real possibility, and the limiting step for the successful prediction of protein interactions is the scoring function used to reduce the space of conformations from billions to a few, and eventually one high affinity complex. An atomic level free-energy scoring function that estimates in units of kcal/mol both electrostatic and desolvation interactions (plus van der Waals if appropriate) of protein-protein docked conformations is used to rerank the blind predictions (860 in total) submitted for six targets to the community-wide Critical Assessment of PRediction of Interactions (CAPRI; http://capri.ebi.ac.uk). We found that native-like models often have varying intermolecular contacts and atom clashes, making unlikely that one can construct a universal function that would rank all these models as native-like. Nevertheless, our scoring function is able to consistently identify the native-like complexes as those with the lowest free energy for the individual models of 16 (out of 17) human predictors for five of the targets, while at the same time the modelers failed to do so in more than half of the cases. The scoring of high-quality models developed by a wide variety of methods and force fields confirms that electrostatic and desolvation forces are the dominant interactions determining the bound structure. The CAPRI experiment has shown that modelers can predict valuable models of protein-protein complexes, and improvements in scoring functions should soon solve the docking problem for complexes whose backbones do not change much upon binding. A scoring server and programs are available at http://structure.pitt.edu.     

An overview of data-driven HADDOCK strategies in CAPRI rounds 38-45

Our information-driven docking approach HADDOCK has demonstrated a sustained performance since the start of its participation to CAPRI. This is due, in part, to its ability to integrate data into the modeling process, and to the robustness of its scoring function. We participated in CAPRI both as server and manual predictors. In CAPRI rounds 38-45, we have used various strategies depending on the available information. These ranged from imposing restraints to a few residues identified from literature as being important for the interaction, to binding pockets identified from homologous complexes or template-based refinement/CA-CA restraint-guided docking from identified templates. When relevant, symmetry restraints were used to limit the conformational sampling. We also tested for a large decamer target a new implementation of the MARTINI coarse-grained force field in HADDOCK. Overall, we obtained acceptable or better predictions for 13 and 11 server and manual submissions, respectively, out of the 22 interfaces. Our server performance (acceptable or higher-quality models when considering the top 10) was better (59%) than the manual (50%) one, in which we typically experiment with various combinations of protocols and data sources. Again, our simple scoring function based on a linear combination of intermolecular van der Waals and electrostatic energies and an empirical desolvation term demonstrated a good performance in the scoring experiment with a 63% success rate across all 22 interfaces. An analysis of model quality indicates that, while we are consistently performing well in generating acceptable models, there is room for improvement for generating/identifying higher quality models.     

Structure prediction of biological assemblies using GALAXY in CAPRI rounds 38-45

We participated in CARPI rounds 38-45 both as a server predictor and a human predictor. These CAPRI rounds provided excellent opportunities for testing prediction methods for three classes of protein interactions, that is, protein-protein, protein-peptide, and protein-oligosaccharide interactions. Both template-based methods (GalaxyTBM for monomer protein, GalaxyHomomer for homo-oligomer protein, GalaxyPepDock for protein-peptide complex) and ab initio docking methods (GalaxyTongDock and GalaxyPPDock for protein oligomer, GalaxyPepDock-ab-initio for protein-peptide complex, GalaxyDock2 and Galaxy7TM for protein-oligosaccharide complex) have been tested. Template-based methods depend heavily on the availability of proper templates and template-target similarity, and template-target difference is responsible for inaccuracy of template-based models. Inaccurate template-based models could be improved by our structure refinement and loop modeling methods based on physics-based energy optimization (GalaxyRefineComplex and GalaxyLoop) for several CAPRI targets. Current ab initio docking methods require accurate protein structures as input. Small conformational changes from input structure could be accounted for by our docking methods, producing one of the best models for several CAPRI targets. However, predicting large conformational changes involving protein backbone is still challenging, and full exploration of physics-based methods for such problems is still to come.