Improved statistical methods for hit selection in high-throughput screening

High-throughput screening (HTS) plays a central role in modern drug discovery, allowing the rapid screening of large compound collections against a variety of putative drug targets. HTS is an industrial-scale process, relying on sophisticated automation, control, and state-of-the art detection technologies to organize, test, and measure hundreds of thousands to millions of compounds in nano- to microliter volumes. Despite this high technology, hit selection for HTS is still typically done using simple data analysis and basic statistical methods. The authors discuss in this article some shortcomings of these methods and present alternatives based on modern methods of statistical data analysis. Most important, they describe and show numerous real examples from the biologist-friendly Stat Server HTS application (SHS), a custom-developed software tool built on the commercially available S-PLUS and StatServer statistical analysis and server software. This system remotely processes HTS data using powerful and sophisticated statistical methodology but insulates users from the technical details by outputting results in a variety of readily interpretable graphs and tables.     

A pair of new statistical parameters for quality control in RNA interference high-throughput screening assays

RNA interference (RNAi) high-throughput screening (HTS) enables massive parallel gene silencing and is increasingly being used to reveal novel connections between genes and disease-relevant phenotypes. The application of genome-scale RNAi relies on the development of high-quality RNAi HTS assays. To obtain high-quality HTS assays, there is a strong need for an easily interpretable and theoretically based quality control (QC) metric. Signal-to-noise ratio (S/N), signal-to-background ratio (S/B), and Z-factor have been adopted as QC metrics in HTS assays. In this paper, I proposed a pair of new parameters, strictly standardized mean difference (SSMD) and coefficient of variability in difference (CVD), as QC metrics in RNAi HTS assays. Compared to S/B and S/N, SSMD and CVD capture the variabilities in both compared populations. Compared to Z-factor, SSMD and CVD have a clear probability interpretation and a solid statistical basis. Accordingly, the cutoff criteria of using SSMD or CVD as a QC metric in HTS assays are fully theoretically based. In addition, I discuss the relationship between the SSMD-based criterion and the popular Z-factor-based criterion and elucidate why p-value from t-test of testing mean difference fails to serve as a QC metric.     

MScreen: An Integrated Compound Management and High-Throughput Screening Data Storage and Analysis System

High-throughput screening (HTS) has historically been used by the pharmaceutical industry to rapidly test hundreds of thousands of compounds to identify potential drug candidates. More recently, academic groups have used HTS to identify new chemical probes or small interfering RNA (siRNA) that can serve as experimental tools to examine the biology or physiology of novel proteins, processes, or interactions. HTS presents a significant challenge with the vast and complex nature of data generated. This report describes MScreen, a Web-based, open-source cheminformatics application for chemical library and siRNA plate management, primary HTS and dose-response data handling, structure search, and administrative functions. Each project in MScreen can be secured with passwords or shared in an open-information environment that enables collaborators to easily compare data from many screens, providing a useful means to identify compounds with desired selectivity. Unique features include compound, substance, mixture, and siRNA plate creation and formatting; automated dose-response fitting and quality control (QC); and user, target, and assay method administration. MScreen provides an effective means to facilitate HTS information handling and analysis in the academic setting so that users can efficiently view their screening data and evaluate results for follow-up.     

Prevention,diagnosis and treatment of high‐throughput sequencing data pathologies

High‐throughput sequencing (HTS) technologies generate millions of sequence reads from DNA/RNA molecules rapidly and cost‐effectively, enabling single investigator laboratories to address a variety of ‘omics’ questions in nonmodel organisms, fundamentally changing the way genomic approaches are used to advance biological research. One major challenge posed by HTS is the complexity and difficulty of data quality control (QC). While QC issues associated with sample isolation, library preparation and sequencing are well known and protocols for their handling are widely available, the QC of the actual sequence reads generated by HTS is often overlooked. HTS‐generated sequence reads can contain various errors, biases and artefacts whose identification and amelioration can greatly impact subsequent data analysis. However, a systematic survey on QC procedures for HTS data is still lacking. In this review, we begin by presenting standard ‘health check‐up’ QC procedures recommended for HTS data sets and establishing what ‘healthy’ HTS data look like. We next proceed by classifying errors, biases and artefacts present in HTS data into three major types of ‘pathologies’, discussing their causes and symptoms and illustrating with examples their diagnosis and impact on downstream analyses. We conclude this review by offering examples of successful ‘treatment’ protocols and recommendations on standard practices and treatment options. Notwithstanding the speed with which HTS technologies – and consequently their pathologies – change, we argue that careful QC of HTS data is an important – yet often neglected – aspect of their application in molecular ecology, and lay the groundwork for developing a HTS data QC ‘best practices’ guide.     

Identifying actives from HTS data sets: practical approaches for the selection of an appropriate HTS data-processing method and quality control review

High-throughput screening (HTS) has achieved a dominant role in drug discovery over the past 2 decades. The goal of HTS is to identify active compounds (hits) by screening large numbers of diverse chemical compounds against selected targets and/or cellular phenotypes. The HTS process consists of multiple automated steps involving compound handling, liquid transfers, and assay signal capture, all of which unavoidably contribute to systematic variation in the screening data. The challenge is to distinguish biologically active compounds from assay variability. Traditional plate controls-based and non-controls-based statistical methods have been widely used for HTS data processing and active identification by both the pharmaceutical industry and academic sectors. More recently, improved robust statistical methods have been introduced, reducing the impact of systematic row/column effects in HTS data. To apply such robust methods effectively and properly, we need to understand their necessity and functionality. Data from 6 HTS case histories are presented to illustrate that robust statistical methods may sometimes be misleading and can result in more, rather than less, false positives or false negatives. In practice, no single method is the best hit detection method for every HTS data set. However, to aid the selection of the most appropriate HTS data-processing and active identification methods, the authors developed a 3-step statistical decision methodology. Step 1 is to determine the most appropriate HTS data-processing method and establish criteria for quality control review and active identification from 3-day assay signal window and DMSO validation tests. Step 2 is to perform a multilevel statistical and graphical review of the screening data to exclude data that fall outside the quality control criteria. Step 3 is to apply the established active criterion to the quality-assured data to identify the active compounds.     

A Simple Statistical Parameter for Use in Evaluation and Validation of High Throughput Screening Assays

The ability to identify active compounds (3hits2) from large chemical libraries accurately and rapidly has been the ultimate goal in developing high-throughput screening (HTS) assays. The ability to identify hits from a particular HTS assay depends largely on the suitability or quality of the assay used in the screening. The criteria or parameters for evaluating the 3suitability2 of an HTS assay for hit identification are not well defined and hence it still remains difficult to compare the quality of assays directly. In this report, a screening window coefficient, called 3Z-factor,2 is defined. This coefficient is reflective of both the assay signal dynamic range and the data variation associated with the signal measurements, and therefore is suitable for assay quality assessment. The Z-factor is a dimensionless, simple statistical characteristic for each HTS assay. The Z-factor provides a useful tool for comparison and evaluation of the quality of assays, and can be utilized in assay optimization and validation.     

Use of mixture distributions to deconvolute the behavior of "hits" and controls in high-throughput screening data

The stochastic nature of high-throughput screening (HTS) data indicates that information may be gleaned by applying statistical methods to HTS data. A foundation of parametric statistics is the study and elucidation of population distributions, which can be modeled using modern spreadsheet software. The methods and results described here use fundamental concepts of statistical population distributions analyzed using a spreadsheet to provide tools in a developing armamentarium for extracting information from HTS data. Specific examples using two HTS kinase assays are analyzed. The analyses use normal and gamma distributions, which combine to form mixture distributions. HTS data were found to be described well using such mixture distributions, and deconvolution of the mixtures to the constituent gamma and normal parts provided insight into how the assays performed. In particular, the proportion of hits confirmed was predicted from the original HTS data and used to assess screening assay performance. The analyses also provide a method for determining how hit thresholds--values used to separate active from inactive compounds--affect the proportion of compounds verified as active and how the threshold can be chosen to optimize the selection process.     

Statistical analysis of systematic errors in high-throughput screening

High-throughput screening (HTS) is an efficient technology for drug discovery. It allows for screening of more than 100,000 compounds a day per screen and requires effective procedures for quality control. The authors have developed a method for evaluating a background surface of an HTS assay; it can be used to correct raw HTS data. This correction is necessary to take into account systematic errors that may affect the procedure of hit selection. The described method allows one to analyze experimental HTS data and determine trends and local fluctuations of the corresponding background surfaces. For an assay with a large number of plates, the deviations of the background surface from a plane are caused by systematic errors. Their influence can be minimized by the subtraction of the systematic background from the raw data. Two experimental HTS assays from the ChemBank database are examined in this article. The systematic error present in these data was estimated and removed from them. It enabled the authors to correct the hit selection procedure for both assays.     

The design and development of an integrated natural products screening database

We designed and developed NEXUS--a new natural products screening database and related suite of software applications--to utilize the spectacular increases in assay capacity of the modern high throughput screening (HTS) environment. NEXUS not only supports seamless integration with separate HTS systems, but supports user-customized integration with external laboratory automation, particularly sample preparation systems. Designed and developed based on a detailed process model for natural products drug discovery, NEXUS comprises two integrated parts: (1) a single schema of Oracle tables and callable procedures and functions, and (2) software "front-ends" to the database developed using Microsoft Excel and Oracle Discovery/2000. Many of the back-end processing functions were written in Programming Language/Structured Query Language (PL/SQL) to provide an Application Programmer's Interface, which allows end users to create custom applications with little input from information technology professionals.     

Evaluating real-life high-throughput screening data

High-throughput screening (HTS) is the result of a concerted effort of chemistry, biology, information technology, and engineering. Many factors beyond the biology of the assay influence the quality and outcome of the screening process, yet data analysis and quality control are often focused on the analysis of a limited set of control wells and the calculated values derived from these wells. Taking into account the large number of variables and the amount of data generated, multiple views of the screening data are necessary to guarantee quality and validity of HTS results. This article does not aim to give an exhaustive outlook on HTS data analysis but tries to illustrate the shortfalls of a reductionist approach focused on control wells and give examples for further analysis.     

Bioactivity-guided discovery of quality control markers in rhizomes of Curcuma wenyujin based on spectrum-effect relationship against human lung cancer cells

BackgroundDue to the diversity of the ingredients, the complexity of the mechanism of action, the uncertainty of the effective ingredients, coupled with the multiple species and multiple growing areas, the quality control (QC) of Traditional Chinese Medicines (TCMs) is challenging. Discovering and identifying effective compounds from the complex extracts of TCMs and then establishing a scientific QC method is the key to the holistic QC of TCMs.PurposeTo develop an anti-lung-cancer-guided spectrum-effect relationship approach for the discovery of QC markers of the rhizome of Curcuma wenyujin (WEZ) and establish a bioactive compounds-based holistic QC method.MethodsThe chemical profiling of the volatile oil (WVO) from 42 batches of WEZ collected from different growing areas was performed by GC-MS. The anti-lung cancer activity of different WVO samples was determined by CCK-8 assay against human lung cancer cells (A549). The apoptosis and cell cycle analysis under different concentrations of WVO were detected by flow cytometry. SIMCA-P software was used to perform multivariate statistical analysis on the chemical composition of different WVO samples and to find the different components. Active compounds were screened using a PLSR model of the spectrum-effect relationship. Bioactive compounds-based fingerprint and quantification of the leading bioactive compounds were developed by GC-MS and GC-FID, respectively.ResultsSeventy-eight compounds were detected in WVO and 54 were successfully identified. The multivariate statistical analysis uncovered that WVO components and the anti-A549 activity of WVO at the concentration of 60 nl/ml differ greatly according to the origin of the plant. The WVO at the concentration of 60 nl/ml (IC₅₀) increased A549 cells apoptosis significantly with late and early apoptosis of 15.61% and 7.80%, and the number of cells in the G2/M phase were also increased significantly under this concentration. The spectrum-effect relationship analysis revealed that 44 compounds were positively correlated with their activities, and the result was verified by A549 cell viability assay. Sixteen positively correlated compounds were further selected as QC markers according to their relative amount > 0.5% and anticancer activity. Finally, the 16 QC markers-based GC-MS fingerprint was established to holistically control the quality of WEZ, and a GC-FID method was developed for the quantification of leading bioactive compounds, β-elemene and β-caryophyllene.ConclusionBased on an anti-lung-cancer-guided spectrum-effect relationship approach, the bioactive compounds-based holistic QC method was successfully developed for WEZ, which could provide a valuable reference for the QC of TCMs.     

Novel analytic criteria and effective plate designs for quality control in genome-scale RNAi screens

One of the most fundamental challenges in genome-wide RNA interference (RNAi) screens is to glean biological significance from mounds of data, which relies on the development and adoption of appropriate analytic methods and designs for quality control (QC) and hit selection. Currently, a Z-factor-based QC criterion is widely used to evaluate data quality. However, this criterion cannot take into account the fact that different positive controls may have different effect sizes and leads to inconsistent QC results in experiments with 2 or more positive controls with different effect sizes. In this study, based on a recently proposed parameter, strictly standardized mean difference (SSMD), novel QC criteria are constructed for evaluating data quality in genome-wide RNAi screens. Two good features of these novel criteria are: (1) SSMD has both clear original and probability meanings for evaluating the differentiation between positive and negative controls and hence the SSMD-based QC criteria have a solid probabilistic and statistical basis, and (2) these QC criteria obtain consistent QC results for multiple positive controls with different effect sizes. In addition, I propose multiple plate designs and the guidelines for using them in genome-wide RNAi screens. Finally, I provide strategies for using the SSMD-based QC criteria and effective plate design together to improve data quality. The novel SSMD-based QC criteria, effective plate designs, and related guidelines and strategies may greatly help to obtain high quality of data in genome-wide RNAi screens.     

Determining appropriate substrate conversion for enzymatic assays in high-throughput screening

It is generally accepted that the conversion of substrate should be kept at less than 10% of the total substrate used when studying enzyme kinetics. However, 10% or less substrate conversion often will not produce sufficient signal changes required for robust high-throughput screening (HTS). To increase the signal-to-background ratio, HTS is often performed at higher than 10% substrate conversion. Because the consequences of high substrate conversion are poorly understood, the screening results are sometimes questioned by enzymologists. The quality of an assay is judged by the ability to detect an inhibitor under HTS conditions, which depends on the robustness of the primary detection signal (Z factor) and the sensitivity to an inhibitor. The assay sensitivity to an inhibitor is reflected in the observed IC(50) value or percent inhibition at a fixed compound concentration when single-point data are collected. The major concern for an enzymatic assay under high substrate conversion is that the sensitivity of the screen may be compromised. Here we derive the relationship between the IC(50) value for a given inhibitor and the percentage of substrate conversion using a first-order kinetic model under conditions that obey Henri-Michaelis-Menten kinetics. The derived theory was further verified experimentally with a cAMP-dependent protein kinase. This model provides guidance for assay developers to choose an appropriate substrate conversion in designing an enzymatic assay, balancing the needs for robust signal and sensitivity to inhibitors.     

pkDACLASS: Open source software for analyzing MALDI-TOF data

In recent years, mass spectrometry has become one of the core technologies for high throughput proteomic profiling in biomedical research. However, reproducibility of the results using this technology was in question. It has been realized that sophisticated automatic signal processing algorithms using advanced statistical procedures are needed to analyze high resolution and high dimensional proteomic data, e.g., Matrix-Assisted Laser Desorption/Ionization Time-of-Flight (MALDI-TOF) data. In this paper we present a software package-pkDACLASS based on R which provides a complete data analysis solution for users of MALDITOF raw data. Complete data analysis comprises data preprocessing, monoisotopic peak detection through statistical model fitting and testing, alignment of the monoisotopic peaks for multiple samples and classification of the normal and diseased samples through the detected peaks. The software provides flexibility to the users to accomplish the complete and integrated analysis in one step or conduct analysis as a flexible platform and reveal the results at each and every step of the analysis. AVAILABILITY: The database is available for free at http://cran.r-project.org/web/packages/pkDACLASS/index.html.     

A two-stage, multilevel quality control system for serological assays in anthrax vaccine clinical trials

A two-stage, multilevel assay quality control (QC) system was designed and implemented for two high stringency QC anthrax serological assays; a quantitative anti-PA IgG enzyme-linked immunosorbent assay (ELISA) and an anthrax lethal toxin neutralization activity (TNA) assay. The QC system and the assays were applied for the congressionally mandated Centers for Disease Control and Prevention (CDC) Phase 4 human clinical trial of anthrax vaccine adsorbed (AVA, BioThrax). A total of 57,284 human serum samples were evaluated by anti-PA enzyme-linked immunosorbent assay (ELISA) and 11,685 samples by anthrax lethal toxin neutralization activity (TNA) assay. The QC system demonstrated overall sample acceptance rates of 86% for ELISA and 90% for the TNA assays respectively. Monitoring of multiple assay and test sample variables showed no significant long term trends or degradation in any of the critical assay reagents or reportable values for both assays. Assay quality control data establish the functionality of the quality control system and demonstrates the reliability of the serological data generated using these assays.     

arrayQCplot: software for checking the quality of microarray data

SUMMARY: arrayQCplot is a software for the exploratory analysis of microarray data. This software focuses on quality control and generates newly developed plots for quality and reproducibility checks. It is developed using R and provides a user-friendly graphical interface for graphics and statistical analysis. Therefore, novice users will find arrayQCplot as an easy-to-use software for checking the quality of their data by a simple mouse click. AVAILABILITY: arrayQCplot software is available from Bioconductor at http://www.bioconductor.org. A more detailed manual is available at http://bibs.snu.ac.kr/software/arrayQCplot CONTACT: tspark@stats.snu.ac.kr.     

"Plate cherry picking": a novel semi-sequential screening paradigm for cheaper, faster, information-rich compound selection

This work describes a novel semi-sequential technique for in silico enhancement of high-throughput screening (HTS) experiments now employed at Novartis. It is used in situations in which the size of the screen is limited by the readout (e.g., high-content screens) or the amount of reagents or tools (proteins or cells) available. By performing computational chemical diversity selection on a per plate basis (instead of a per compound basis), 25% of the 1,000,000-compound screening was optimized for general initial HTS. Statistical models are then generated from target-specific primary results (percentage inhibition data) to drive the cherry picking and testing from the entire collection. Using retrospective analysis of 11 HTS campaigns, the authors show that this method would have captured on average two thirds of the active compounds (IC(50) < 10 microM) and three fourths of the active Murcko scaffolds while decreasing screening expenditure by nearly 75%. This result is true for a wide variety of targets, including G-protein-coupled receptors, chemokine receptors, kinases, metalloproteinases, pathway screens, and protein-protein interactions. Unlike time-consuming "classic" sequential approaches that require multiple iterations of cherry picking, testing, and building statistical models, here individual compounds are cherry picked just once, based directly on primary screening data. Strikingly, the authors demonstrate that models built from primary data are as robust as models built from IC(50) data. This is true for all HTS campaigns analyzed, which represent a wide variety of target classes and assay types.     

Design and Implementation of High Throughput Screening Assays

High throughput screening (HTS) is at the core of the drug discovery process, and so it is critical to design and implement HTS assays in a comprehensive fashion involving scientists from the disciplines of biology, chemistry, engineering, and informatics. This requires careful analysis of many variables, starting with the choice of assay target and ending with the discovery of lead compounds. At every step in this process, there are decisions to be made that can greatly impact the outcome of the HTS effort, to the point of making it a success or a failure. Although specific guidelines should be established to insure that the screening assay reaches an acceptable level of quality, many choices require pragmatism and the ability to compromise opposing forces.     

A novel specific edge effect correction method for RNA interference screenings

MOTIVATION: High-throughput screening (HTS) is an important method in drug discovery in which the activities of a large number of candidate chemicals or genetic materials are rapidly evaluated. Data are usually obtained by measurements on samples in microwell plates and are often subjected to artefacts that can bias the result selection. We report here a novel edge effect correction algorithm suitable for RNA interference (RNAi) screening, because its normalization does not rely on the entire dataset and takes into account the specificities of such a screening process. The proposed method is able to estimate the edge effects for each assay plate individually using the data from a single control column based on diffusion model, and thus targeting a specific but recurrent well-known HTS artefact. This method was first developed and validated using control plates and was then applied to the correction of experimental data generated during a genome-wide siRNA screen aimed at studying HIV-host interactions. The proposed algorithm was able to correct the edge effect biasing the control data and thus improve assay quality and, consequently, the hit-selection step.     

Systematic error detection in experimental high-throughput screening

Background  

High-throughput screening (HTS) is a key part of the drug discovery process during which thousands of chemical compounds are screened and their activity levels measured in order to identify potential drug candidates (i.e., hits). Many technical, procedural or environmental factors can cause systematic measurement error or inequalities in the conditions in which the measurements are taken. Such systematic error has the potential to critically affect the hit selection process. Several error correction methods and software have been developed to address this issue in the context of experimental HTS [1–7]. Despite their power to reduce the impact of systematic error when applied to error perturbed datasets, those methods also have one disadvantage - they introduce a bias when applied to data not containing any systematic error [6]. Hence, we need first to assess the presence of systematic error in a given HTS assay and then carry out systematic error correction method if and only if the presence of systematic error has been confirmed by statistical tests.