An efficient method for the detection and elimination of systematic error in high-throughput screening

MOTIVATION: High-throughput screening (HTS) is an early-stage process in drug discovery which allows thousands of chemical compounds to be tested in a single study. We report a method for correcting HTS data prior to the hit selection process (i.e. selection of active compounds). The proposed correction minimizes the impact of systematic errors which may affect the hit selection in HTS. The introduced method, called a well correction, proceeds by correcting the distribution of measurements within wells of a given HTS assay. We use simulated and experimental data to illustrate the advantages of the new method compared to other widely-used methods of data correction and hit selection in HTS. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.     

Assay concordance between SPA and TR-FRET in high-throughput screening

High-throughput screening (HTS) of large chemical libraries has become the main source of new lead compounds for drug development. Several specialized detection technologies have been developed to facilitate the cost- and time-efficient screening of millions of compounds. However, concerns have been raised, claiming that different HTS technologies may produce different hits, thus limiting trust in the reliability of HTS data. This study was aimed to investigate the reliability of the authors most frequently used assay techniques: scintillation proximity assay (SPA) and homogeneous time-resolved fluorescence resonance energy transfer (TR-FRET). To investigate the data concordance between these 2 detection technologies, the authors screened a large subset of the Schering compound library consisting of 300,000 compounds for inhibitors of a nonreceptor tyrosine kinase. They chose to set up this study in realistic HTS scale to ensure statistical significance of the results. The findings clearly demonstrate that the choice of detection technology has no significant impact on hit finding, provided that assays are biochemically equivalent. Data concordance is up to 90%. The little differences in hit findings are caused by threshold setting but not by systematic differences between the technologies. The most significant difference between the compared techniques is that in the SPA format, more false-positive primary hits were obtained.     

A new method with flexible and balanced control of false negatives and false positives for hit selection in RNA interference high-throughput screening assays

The z-score method and its variants for testing mean difference are commonly used for hit selection in high-throughput screening (HTS) assays. Strictly standardized mean difference (SSMD) offers a way to measure and classify the short interfering RNA (siRNA) effects. In this article, based on SSMD, the authors propose a new testing method for hit selection in RNA interference (RNAi) HTS assays. This SSMD-based method allows the differentiation between siRNAs with large and small effects on the assay output and maintains flexible and balanced control of both the false-negative rate, in which the siRNAs with strong effects are not selected as hits, and the restricted false-positive rate, in which the siRNAs with weak or no effects are selected as hits. This method directly addresses the size of siRNA effects represented by the strength of difference between an siRNA and a negative reference, whereas the classic z-score method and t-test of testing no mean difference address whether the mean of an siRNA is exactly the same as the mean of a negative reference. This method can readily control the false-negative rate, whereas it is nontrivial for the classic z-score method and t-test to control the false-negative rate. Therefore, theoretically, the SSMD-based method offers better control of the sizes of siRNA effects and the associated false-positive and false-negative rates than the commonly used z-score method and t-test for hit selection in HTS assays. The SSMD-based method should generally be applicable to any assay in which the end point is a difference in signal compared to a reference sample, including those for RNAi, receptor, enzyme, and cellular function.     

Probing the primary screening efficiency by multiple replicate testing: a quantitative analysis of hit confirmation and false screening results of a biochemical assay

Despite a large body of references on assay development, assay optimization, strategies, and methodologies for high-throughput screening (HTS), there have been few reports on investigations of the efficiency of primary screening in a systematic and quantitative manner for a typical HTS process. Recently, the authors investigated the primary hit comparison and the effect of measurement variability by screening a library of approximately 25,000 random compounds in multiple replicate tests in a nuclear receptor recruitment assay with 2 different assay detection technologies. In this report, we utilized these sets of multiple replicate screening data from a different perspective and conducted a systematic data analysis in order to gain some insights into the hit-finding efficiency of a typical primary screening process. Specifically, hit confirmation, false-positive (declaration) rates, and false-negative rates at different hit cutoff limits were explored and calculated from the 2 different assay formats. Results and analyses provided some quantitative estimation regarding the reliability and efficiency of the primary screening process. For the 2 assay formats tested in this report, the confirmation rate (activity repeated at or above a certain hit limit) was found to be 65% or above. It was also suggested that, at least in this case, applying some hit-selection strategies, it is possible to decrease the number of false-negative or false-positive hits without significantly increasing the efforts in primary screening.     

Identifying actives from HTS data sets: practical approaches for the selection of an appropriate HTS data-processing method and quality control review

High-throughput screening (HTS) has achieved a dominant role in drug discovery over the past 2 decades. The goal of HTS is to identify active compounds (hits) by screening large numbers of diverse chemical compounds against selected targets and/or cellular phenotypes. The HTS process consists of multiple automated steps involving compound handling, liquid transfers, and assay signal capture, all of which unavoidably contribute to systematic variation in the screening data. The challenge is to distinguish biologically active compounds from assay variability. Traditional plate controls-based and non-controls-based statistical methods have been widely used for HTS data processing and active identification by both the pharmaceutical industry and academic sectors. More recently, improved robust statistical methods have been introduced, reducing the impact of systematic row/column effects in HTS data. To apply such robust methods effectively and properly, we need to understand their necessity and functionality. Data from 6 HTS case histories are presented to illustrate that robust statistical methods may sometimes be misleading and can result in more, rather than less, false positives or false negatives. In practice, no single method is the best hit detection method for every HTS data set. However, to aid the selection of the most appropriate HTS data-processing and active identification methods, the authors developed a 3-step statistical decision methodology. Step 1 is to determine the most appropriate HTS data-processing method and establish criteria for quality control review and active identification from 3-day assay signal window and DMSO validation tests. Step 2 is to perform a multilevel statistical and graphical review of the screening data to exclude data that fall outside the quality control criteria. Step 3 is to apply the established active criterion to the quality-assured data to identify the active compounds.     

A Simple Statistical Parameter for Use in Evaluation and Validation of High Throughput Screening Assays

The ability to identify active compounds (3hits2) from large chemical libraries accurately and rapidly has been the ultimate goal in developing high-throughput screening (HTS) assays. The ability to identify hits from a particular HTS assay depends largely on the suitability or quality of the assay used in the screening. The criteria or parameters for evaluating the 3suitability2 of an HTS assay for hit identification are not well defined and hence it still remains difficult to compare the quality of assays directly. In this report, a screening window coefficient, called 3Z-factor,2 is defined. This coefficient is reflective of both the assay signal dynamic range and the data variation associated with the signal measurements, and therefore is suitable for assay quality assessment. The Z-factor is a dimensionless, simple statistical characteristic for each HTS assay. The Z-factor provides a useful tool for comparison and evaluation of the quality of assays, and can be utilized in assay optimization and validation.     

Use of mixture distributions to deconvolute the behavior of "hits" and controls in high-throughput screening data

The stochastic nature of high-throughput screening (HTS) data indicates that information may be gleaned by applying statistical methods to HTS data. A foundation of parametric statistics is the study and elucidation of population distributions, which can be modeled using modern spreadsheet software. The methods and results described here use fundamental concepts of statistical population distributions analyzed using a spreadsheet to provide tools in a developing armamentarium for extracting information from HTS data. Specific examples using two HTS kinase assays are analyzed. The analyses use normal and gamma distributions, which combine to form mixture distributions. HTS data were found to be described well using such mixture distributions, and deconvolution of the mixtures to the constituent gamma and normal parts provided insight into how the assays performed. In particular, the proportion of hits confirmed was predicted from the original HTS data and used to assess screening assay performance. The analyses also provide a method for determining how hit thresholds--values used to separate active from inactive compounds--affect the proportion of compounds verified as active and how the threshold can be chosen to optimize the selection process.     

Integrating experimental and analytic approaches to improve data quality in genome-wide RNAi screens

RNA interference (RNAi) not only plays an important role in drug discovery but can also be developed directly into drugs. RNAi high-throughput screening (HTS) biotechnology allows us to conduct genome-wide RNAi research. A central challenge in genome-wide RNAi research is to integrate both experimental and computational approaches to obtain high quality RNAi HTS assays. Based on our daily practice in RNAi HTS experiments, we propose the implementation of 3 experimental and analytic processes to improve the quality of data from RNAi HTS biotechnology: (1) select effective biological controls; (2) adopt appropriate plate designs to display and/or adjust for systematic errors of measurement; and (3) use effective analytic metrics to assess data quality. The applications in 5 real RNAi HTS experiments demonstrate the effectiveness of integrating these processes to improve data quality. Due to the effectiveness in improving data quality in RNAi HTS experiments, the methods and guidelines contained in the 3 experimental and analytic processes are likely to have broad utility in genome-wide RNAi research.     

HTS promiscuity analyses for accelerating decision making

Frequent hitters are compounds that are detected as a "hit" in multiple high-throughput screening (HTS) assays. Such behavior is specific (e.g., target family related) or unspecific (e.g., reactive compounds) or can result from a combination of such behaviors. Detecting such hits while predicting the underlying reason behind their promiscuous behavior is desirable because it provides valuable information not only about the compounds themselves but also about the assay methodology and target classes at hand. This information can also greatly reduce cost and time during HTS hit profiling. The present study exemplifies how to mine large HTS data repositories, such as the one at Boehringer Ingelheim, to identify frequent hitters, gain further insights into the causes of promiscuous behavior, and generate models for predicting promiscuous compounds. Applications of this approach are demonstrated using two recent large-scale HTS assays. The authors believe this analysis and its concrete applications are valuable tools for streamlining and accelerating decision-making processes during the course of hit discovery.     

Quantitative assessment of hit detection and confirmation in single and duplicate high-throughput screenings

The process of identifying active targets (hits) in high-throughput screening (HTS) usually involves 2 steps: first, removing or adjusting for systematic variation in the measurement process so that extreme values represent strong biological activity instead of systematic biases such as plate effect or edge effect and, second, choosing a meaningful cutoff on the calculated statistic to declare positive compounds. Both false-positive and false-negative errors are inevitable in this process. Common control or estimation of error rates is often based on an assumption of normal distribution of the noise. The error rates in hit detection, especially false-negative rates, are hard to verify because in most assays, only compounds selected in primary screening are followed up in confirmation experiments. In this article, the authors take advantage of a quantitative HTS experiment in which all compounds are tested 42 times over a wide range of 14 concentrations so true positives can be found through a dose-response curve. Using the activity status defined by dose curve, the authors analyzed the effect of various data-processing procedures on the sensitivity and specificity of hit detection, the control of error rate, and hit confirmation. A new summary score is proposed and demonstrated to perform well in hit detection and useful in confirmation rate estimation. In general, adjusting for positional effects is beneficial, but a robust test can prevent overadjustment. Error rates estimated based on normal assumption do not agree with actual error rates, for the tails of noise distribution deviate from normal distribution. However, false discovery rate based on empirically estimated null distribution is very close to observed false discovery proportion.     

Median absolute deviation to improve hit selection for genome-scale RNAi screens

High-throughput screening (HTS) of large-scale RNA interference (RNAi) libraries has become an increasingly popular method of functional genomics in recent years. Cell-based assays used for RNAi screening often produce small dynamic ranges and significant variability because of the combination of cellular heterogeneity, transfection efficiency, and the intrinsic nature of the genes being targeted. These properties make reliable hit selection in the RNAi screen a difficult task. The use of robust methods based on median and median absolute deviation (MAD) has been suggested to improve hit selection in such cases, but mean and standard deviation (SD)-based methods are still predominantly used in many RNAi HTS. In an experimental approach to compare these 2 methods, a genome-scale small interfering RNA (siRNA) screen was performed, in which the identification of novel targets increasing the therapeutic index of the chemotherapeutic agent mitomycin C (MMC) was sought. MAD values were resistant to the presence of outliers, and the hits selected by the MAD-based method included all the hits that would be selected by SD-based method as well as a significant number of additional hits. When retested in triplicate, a similar percentage of these siRNAs were shown to genuinely sensitize cells to MMC compared with the hits shared between SD- and MAD-based methods. Confirmed hits were enriched with the genes involved in the DNA damage response and cell cycle regulation, validating the overall hit selection strategy. Finally, computer simulations showed the superiority and generality of the MAD-based method in various RNAi HTS data models. In conclusion, the authors demonstrate that the MAD-based hit selection method rescued physiologically relevant false negatives that would have been missed in the SD-based method, and they believe it to be the desirable 1st-choice hit selection method for RNAi screen results.     

HTS-Corrector: software for the statistical analysis and correction of experimental high-throughput screening data

MOTIVATION: High-throughput screening (HTS) plays a central role in modern drug discovery, allowing for testing of >100,000 compounds per screen. The aim of our work was to develop and implement methods for minimizing the impact of systematic error in the analysis of HTS data. To the best of our knowledge, two new data correction methods included in HTS-Corrector are not available in any existing commercial software or freeware. RESULTS: This paper describes HTS-Corrector, a software application for the analysis of HTS data, detection and visualization of systematic error, and corresponding correction of HTS signals. Three new methods for the statistical analysis and correction of raw HTS data are included in HTS-Corrector: background evaluation, well correction and hit-sigma distribution procedures intended to minimize the impact of systematic errors. We discuss the main features of HTS-Corrector and demonstrate the benefits of the algorithms.     

Comparison of miniaturized time-resolved fluorescence resonance energy transfer and enzyme-coupled luciferase high-throughput screening assays to discover inhibitors of Rho-kinase II (ROCK-II)

Kinases are important drug discovery targets for a wide variety of therapeutic indications; consequently, the measurement of kinase activity remains a common high-throughput screening (HTS) application. Recently, enzyme-coupled luciferase-kinase (LK) format assays have been introduced. This format measures luminescence resulting from metabolism of adenosine triphosphate (ATP) via a luciferin/luciferase-coupled reaction. In the research presented here, 1536-well format time-resolved fluorescence resonance energy transfer (TR-FRET) and LK assays were created to identify novel Rho-associated kinase II (ROCK-II) inhibitors. HTS campaigns for both assays were conducted in this miniaturized format. It was found that both assays were able to consistently reproduce the expected pharmacology of inhibitors known to be specific to ROCK-II (fasudil IC50: 283 +/- 27 nM and 336 +/- 54 nM for TR-FRET and LK assays, respectively; Y-27632 IC50: 133 +/- 7.8 nM and 150 +/- 22 nM for TR-FRET and LK assays, respectively). In addition, both assays proved robust for HTS efforts, demonstrating excellent plate Z' values during the HTS campaign (0.84 +/- 0.03; 0.72 +/- 0.05 for LK and TR-FRET campaigns, respectively). Both formats identified scaffolds of known and novel ROCK-II inhibitors with similar sensitivity. A comparison of the performance of these 2 assay formats in an HTS campaign was enabled by the existence of a subset of 25,000 compounds found in both our institutional and the Molecular Library Screening Center Network screening files. Analysis of the HTS campaign results based on this subset of common compounds showed that both formats had comparable total hit rates, hit distributions, amount of hit clusters, and format-specific artifact. It can be concluded that both assay formats are suitable for the discovery of ROCK-II inhibitors, and the choice of assay format depends on reagents and/or screening technology available.     

Three mechanistically distinct kinase assays compared: Measurement of intrinsic ATPase activity identified the most comprehensive set of ITK inhibitors

Numerous assay methods have been developed to identify small-molecule effectors of protein kinases, but no single method can be applied to all isolated kinases. The authors developed a set of 3 high-throughput screening (HTS)-compatible biochemical assays that can measure 3 mechanistically distinct properties of a kinase active site, with the goal that at least 1 of the 3 would be applicable to any kinase selected as a target for drug discovery efforts. Two assays measure catalytically active enzyme: A dissociation-enhanced lanthanide fluoroimmuno assay (DELFIA) uses an antibody to quantitate the generation of phosphorylated substrate; a second assay uses luciferase to measure the consumption of adenosine triphosphate (ATP) during either phosphoryl-transfer to a peptide substrate or to water (intrinsic ATPase activity). A third assay, which is not dependent on a catalytically active enzyme, measures the competition for binding to kinase between an inhibitor and a fluorescent ATP binding site probe. To evaluate the suitability of these assays for drug discovery, the authors compared their ability to identify inhibitors of a nonreceptor protein tyrosine kinase from the Tec family, interleukin-2-inducible T cell kinase (ITK). The 3 assays agreed on 57% of the combined confirmed hit set identified from screening a 10,208-compound library enriched with known kinase inhibitors and molecules that were structurally similar. Among the 3 assays, the one measuring intrinsic ATPase activity produced the largest number of unique hits, the fewest unique misses, and the most comprehensive hit set, missing only 2.7% of the confirmed inhibitors identified by the other 2 assays combined. Based on these data, all 3 assay formats are viable for screening and together provide greater options for assay design depending on the targeted kinase.     

Systematic error detection in experimental high-throughput screening

Background  

High-throughput screening (HTS) is a key part of the drug discovery process during which thousands of chemical compounds are screened and their activity levels measured in order to identify potential drug candidates (i.e., hits). Many technical, procedural or environmental factors can cause systematic measurement error or inequalities in the conditions in which the measurements are taken. Such systematic error has the potential to critically affect the hit selection process. Several error correction methods and software have been developed to address this issue in the context of experimental HTS [1–7]. Despite their power to reduce the impact of systematic error when applied to error perturbed datasets, those methods also have one disadvantage - they introduce a bias when applied to data not containing any systematic error [6]. Hence, we need first to assess the presence of systematic error in a given HTS assay and then carry out systematic error correction method if and only if the presence of systematic error has been confirmed by statistical tests.     

Multidimensional profiling of CSF1R screening hits and inhibitors: assessing cellular activity, target residence time, and selectivity in a higher throughput way

Over the past years, improvements in high-throughput screening (HTS) technology and compound libraries have resulted in a dramatic increase in the amounts of good-quality screening hits, and there is a growing need for follow-on hit profiling assays with medium throughput to further triage hits. Here the authors present such assays for the colony-stimulating factor 1 receptor (CSF1R, Fms), including tests for cellular activity and a homogeneous assay to measure affinity for inactive CSF1R. They also present a high-throughput assay to measure target residence time, which is based on competitive binding kinetics. To better fit k(off) rates, they present a modified mathematical model for competitive kinetics. In all assays, they profiled eight reference inhibitors (imatinib, sorafenib, sunitinib, tandutinib, dasatinib, GW2580, Ki20227, and J&J's pyrido[2,3-d]pyrimidin-5-one). Using the known biochemical selectivities of these inhibitors, which can be quantified using metrics such as the selectivity entropy, the authors have determined which assay readout best predicts hit selectivity. Their profiling shows surprisingly that imatinib has a preference for the active form of CSF1R and that Ki20227 has an unusually slow target dissociation rate. This confirms that follow-on hit profiling is essential to ensure that the best hits are selected for lead optimization.     

Robust hit identification by quality assurance and multivariate data analysis of a high-content, cell-based assay

Recent technological advances in high-content screening instrumentation have increased its ease of use and throughput, expanding the application of high-content screening to the early stages of drug discovery. However, high-content screens produce complex data sets, presenting a challenge for both extraction and interpretation of meaningful information. This shifts the high-content screening process bottleneck from the experimental to the analytical stage. In this article, the authors discuss different approaches of data analysis, using a phenotypic neurite outgrowth screen as an example. Distance measurements and hierarchical clustering methods lead to a profound understanding of different high-content screening readouts. In addition, the authors introduce a hit selection procedure based on machine learning methods and demonstrate that this method increases the hit verification rate significantly (up to a factor of 5), compared to conventional hit selection based on single readouts only.     

SpeedScreen: The "missing link" between genomics and lead discovery

SpeedScreen is a novel, label-free, in-solution, affinity-based selection methodology for high-throughput screening (HTS) developed at Novartis Pharma. The SpeedScreen protocol comprises in-solution affinity selection, followed by size exclusion chromatography in combination with microbore-liquid-chromatography/electrospray-ionization mass spectrometry (micro-LC/ESI-MS). The authors describe the basic concept behind assay development, HTS, and data analysis with the SpeedScreen technology. Advantages and limitations of SpeedScreen compared to alternative screening technologies are discussed, and an example is given from a SpeedScreen campaign applying this innovative affinity selection concept in HTS.     

The use of strictly standardized mean difference for hit selection in primary RNA interference high-throughput screening experiments

RNA interference (RNAi) high-throughput screening (HTS) has been hailed as the 2nd genomics wave following the 1st genomics wave of gene expression microarrays and single-nucleotide polymorphism discovery platforms. Following an RNAi HTS, the authors are interested in identifying short interfering RNA (siRNA) hits with large inhibition/activation effects. For hit selection, the z-score method and its variants are commonly used in primary RNAi HTS experiments. Recently, strictly standardized mean difference (SSMD) has been proposed to measure the siRNA effect represented by the magnitude of difference between an siRNA and a negative reference group. The links between SSMD and d+-probability offer a clear interpretation of siRNA effects from a probability perspective. Hence, SSMD can be used as a ranking metric for hit selection. In this article, the authors investigated both the SSMD-based testing process and the use of SSMD as a ranking metric for hit selection in 2 primary siRNA HTS experiments. The analysis results showed that, as a ranking metric, SSMD was more stable and reliable than percentage inhibition and led to more robust hit selection results. Using the SSMD -based testing method, the false-negative rate can more readily be obtained. More important, the use of the SSMD-based method can result in a reduction in both the false-negative and false-positive rates. The applications presented in this article demonstrate that the SSMD method addresses scientific questions and fills scientific needs better than both percentage inhibition and the commonly used z-score method for hit selection.     

Oncology drug discovery applications using the FMAT 8100 HTS system

High-throughput screening (HTS) for potential anticancer agents requires a broad portfolio of assay platforms that may include kinase enzyme assays, protein-protein binding assays, and functional cell-based apoptosis assays. The authors have explored the use of fluorometric microvolume assay technology (the FMAT 8100 HTS System) in three distinct homogeneous HTS assays: (1). a Src tyrosine kinase enzyme assay, (2). a Grb2-SH2 protein-peptide interaction assay, and (3). an annexin V binding apoptosis assay. Data obtained from all three assays suggest that the FMAT system should facilitate the implementation of homogeneous assays for a wide variety of molecular targeted and cell-based screens.