Reproducibility of microarray data: a further analysis of microarray quality control (MAQC) data

Background  

Many researchers are concerned with the comparability and reliability of microarray gene expression data. Recent completion of the MicroArray Quality Control (MAQC) project provides a unique opportunity to assess reproducibility across multiple sites and the comparability across multiple platforms. The MAQC analysis presented for the conclusion of inter- and intra-platform comparability/reproducibility of microarray gene expression measurements is inadequate. We evaluate the reproducibility/comparability of the MAQC data for 12901 common genes in four titration samples generated from five high-density one-color microarray platforms and the TaqMan technology. We discuss some of the problems with the use of correlation coefficient as metric to evaluate the inter- and intra-platform reproducibility and the percent of overlapping genes (POG) as a measure for evaluation of a gene selection procedure by MAQC.     

The MicroArray Quality Control (MAQC) project shows inter- and intraplatform reproducibility of gene expression measurements

Over the last decade, the introduction of microarray technology has had a profound impact on gene expression research. The publication of studies with dissimilar or altogether contradictory results, obtained using different microarray platforms to analyze identical RNA samples, has raised concerns about the reliability of this technology. The MicroArray Quality Control (MAQC) project was initiated to address these concerns, as well as other performance and data analysis issues. Expression data on four titration pools from two distinct reference RNA samples were generated at multiple test sites using a variety of microarray-based and alternative technology platforms. Here we describe the experimental design and probe mapping efforts behind the MAQC project. We show intraplatform consistency across test sites as well as a high level of interplatform concordance in terms of genes identified as differentially expressed. This study provides a resource that represents an important first step toward establishing a framework for the use of microarrays in clinical and regulatory settings.     

A sequence-oriented comparison of gene expression measurements across different hybridization-based technologies

Over the last decade, gene expression microarrays have had a profound impact on biomedical research. The diversity of platforms and analytical methods available to researchers have made the comparison of data from multiple platforms challenging. In this study, we describe a framework for comparisons across platforms and laboratories. We have attempted to include nearly all the available commercial and 'in-house' platforms. Using probe sequences matched at the exon level improved consistency of measurements across the different microarray platforms compared to annotation-based matches. Generally, consistency was good for highly expressed genes, and variable for genes with lower expression values as confirmed by quantitative real-time (QRT)-PCR. Concordance of measurements was higher between laboratories on the same platform than across platforms. We demonstrate that, after stringent preprocessing, commercial arrays were more consistent than in-house arrays, and by most measures, one-dye platforms were more consistent than two-dye platforms.     

Comparison of gene expression measurements from cDNA and 60-mer oligonucleotide microarrays

As the data generated by microarray technology continue to amass, it is necessary to compare and combine gene expression data from different platforms. To evaluate the performance of cDNA and long oligonucleotide (60-mer) arrays, we generated gene expression profiles for two cancer cell lines and compared the data between the two platforms. All 6182 unique genes represented on both platforms were included in the analysis. A limited correlation (r = 0.4708) was obtained and the difference in measurement of low-expression genes was considered to contribute to the limited correlation. Further restriction of the data set to differentially expressed genes detected in cDNA microarrays (1205 genes) and oligonucleotide arrays (1325 genes) showed modest correlations of 0.7076 and 0.6441 between the two platforms. Quantitative real-time PCR measurements of a set of 10 genes showed better correlation with oligonucleotide arrays. Our results demonstrate that there is substantial variation in the data generated from cDNA and 60-mer oligonucleotide arrays. Although general agreement was observed in measurements of differentially expressed genes, we suggest that data from different platforms could not be directly amassed.     

A semi-parametric statistical model for integrating gene expression profiles across different platforms

Determining differentially expressed genes (DEGs) between biological samples is the key to understand how genotype gives rise to phenotype. RNA-seq and microarray are two main technologies for profiling gene expression levels. However, considerable discrepancy has been found between DEGs detected using the two technologies. Integration data across these two platforms has the potential to improve the power and reliability of DEG detection. We propose a rank-based semi-parametric model to determine DEGs using information across different sources and apply it to the integration of RNA-seq and microarray data. By incorporating both the significance of differential expression and the consistency across platforms, our method effectively detects DEGs with moderate but consistent signals. We demonstrate the effectiveness of our method using simulation studies, MAQC/SEQC data and a synthetic microRNA dataset. Our integration method is not only robust to noise and heterogeneity in the data, but also adaptive to the structure of data. In our simulations and real data studies, our approach shows a higher discriminate power and identifies more biologically relevant DEGs than eBayes, DEseq and some commonly used meta-analysis methods.     

Universal external RNA controls for microbial gene expression analysis using microarray and qRT-PCR

Gene expression analysis provides significant insight to understand regulatory mechanisms of biology, yet acquisition and reproduction of quality data, as well as data confirmation and verification remain challenging due to a lack of proper quality controls across different assay platforms. We present a set of six universal external RNA quality controls for microbial mRNA expression analysis that can be applied to both DNA oligo microarray and real-time qRT-PCR including using SYBR Green and TaqMan probe-based chemistry. This set of controls was applied for Saccharomyces cerevisiae and Pseudomonas fluorescens Pf-5 microarray assays and qRT-PCR for yeast gene expression analysis. Highly fitted linear relationships between detected signal intensity and mRNA input were described. Valid mRNA detection range, from 10 to 7000 pg and from 100 fg to 1000 pg were defined for microarray and qRT-PCR assay, respectively. Quantitative estimation of mRNA abundance was tested using randomly selected yeast ORF including function unknown genes using the same source of samples by the two assay platforms. Estimates of mRNA abundance by the two methods were similar and highly correlated in an overlapping detection range from 10 to 1000 pg. The universal external RNA controls provide a means to compare microbial gene expression data derived from different experiments and different platforms for verification and confirmation. Such quality controls ensure reliability and reproducibility of gene expression data, and provide unbiased normalization reference for validation, quantification, and estimate of variation of gene expression experiments. Application of these controls also improves efficiency and facilitates high throughput applications of gene expression analysis using the qRT-PCR assay.     

Rat toxicogenomic study reveals analytical consistency across microarray platforms

To validate and extend the findings of the MicroArray Quality Control (MAQC) project, a biologically relevant toxicogenomics data set was generated using 36 RNA samples from rats treated with three chemicals (aristolochic acid, riddelliine and comfrey) and each sample was hybridized to four microarray platforms. The MAQC project assessed concordance in intersite and cross-platform comparisons and the impact of gene selection methods on the reproducibility of profiling data in terms of differentially expressed genes using distinct reference RNA samples. The real-world toxicogenomic data set reported here showed high concordance in intersite and cross-platform comparisons. Further, gene lists generated by fold-change ranking were more reproducible than those obtained by t-test P value or Significance Analysis of Microarrays. Finally, gene lists generated by fold-change ranking with a nonstringent P-value cutoff showed increased consistency in Gene Ontology terms and pathways, and hence the biological impact of chemical exposure could be reliably deduced from all platforms analyzed.     

Performance comparison of one-color and two-color platforms within the MicroArray Quality Control (MAQC) project

Microarray-based expression profiling experiments typically use either a one-color or a two-color design to measure mRNA abundance. The validity of each approach has been amply demonstrated. Here we provide a simultaneous comparison of results from one- and two-color labeling designs, using two independent RNA samples from the Microarray Quality Control (MAQC) project, tested on each of three different microarray platforms. The data were evaluated in terms of reproducibility, specificity, sensitivity and accuracy to determine if the two approaches provide comparable results. For each of the three microarray platforms tested, the results show good agreement with high correlation coefficients and high concordance of differentially expressed gene lists within each platform. Cumulatively, these comparisons indicate that data quality is essentially equivalent between the one- and two-color approaches and strongly suggest that this variable need not be a primary factor in decisions regarding experimental microarray design.     

Correlation analysis of external RNA controls reveals its utility for assessment of microarray assay

Quality control of a microarray experiment has become an important issue for both research and regulation. External RNA controls (ERCs), which can be either added to the total RNA level (tERCs) or introduced right before hybridization (cERCs), are designed and recommended by commercial microarray platforms for assessment of performance of a microarray experiment. However, the utility of ERCs has not been fully realized mainly due to the lack of sufficient data resources. The US Food and Drug Administration (FDA)-led community-wide Microarray Quality Control (MAQC) study generates a large amount of microarray data with implementation of ERCs across several commercial microarray platforms. The utility of ERCs in quality control by assessing the ERCs’ concentration-response behavior was investigated in the MAQC study. In this work, an ERC-based correlation analysis was conducted to assess the quality of a microarray experiment. We found that the pairwise correlations of tERCs are sample independent, indicating that the array data obtained from different biological samples can be treated as technical replicates in analysis of tERCs. Consequently, the commonly used quality control method of applying correlation analysis on technical replicates can be adopted for assessing array performance based on different biological samples using tERCs. The proposed approach is sensitive to identifying outlying assays and is not dependent on the choice of normalization method.     

Differences in gene expression between B-cell chronic lymphocytic leukemia and normal B cells: a meta-analysis of three microarray studies

MOTIVATION: A major focus of current cancer research is to identify genes that can be used as markers for prognosis and diagnosis, and as targets for therapy. Microarray technology has been applied extensively for this purpose, even though it has been reported that the agreement between microarray platforms is poor. A critical question is: how can we best combine the measurements of matched genes across microarray platforms to develop diagnostic and prognostic tools related to the underlying biology? RESULTS: We introduce a statistical approach within a Bayesian framework to combine the microarray data on matched genes from three investigations of gene expression profiling of B-cell chronic lymphocytic leukemia (CLL) and normal B cells (NBC) using three different microarray platforms, oligonucleotide arrays, cDNA arrays printed on glass slides and cDNA arrays printed on nylon membranes. Using this approach, we identified a number of genes that were consistently differentially expressed between CLL and NBC samples.     

Arrays of hope

A large study by the MAQC consortium has established that different DNA microarray platforms can generate reproducible lists of differentially expressed genes. Now scientists are grappling with the challenges of moving the technology toward the clinic.     

An integrated cross-platform prognosis study on neuroblastoma patients

There have been several reports about the potential for predicting prognosis of neuroblastoma patients using microarray gene expression profiling of the tumors. However these studies have revealed an apparent diversity in the identity of the genes in their predictive signatures. To test the contribution of the platform to this discrepancy we applied the z-scoring method to minimize the impact of platform and combine gene expression profiles of neuroblastoma (NB) tumors from two different platforms, cDNA and Affymetrix. A total of 12442 genes were common to both cDNA and Affymetrix arrays in our data set. Two-way ANOVA analysis was applied to the combined data set for assessing the relative effect of prognosis and platform on gene expression. We found that 26.6% (3307) of the genes had significant impact on survival. There was no significant impact of microarray platform on expression after application of z-scoring standardization procedure. Artificial neural network (ANN) analysis of the combined data set in a leave-one-out prediction strategy correctly predicted the outcome for 90% of the samples. Hierarchical clustering analysis using the top-ranked 160 genes showed the great separation of two clusters, and the majority of matched samples from the different platforms were clustered next to each other. The ANN classifier trained with our combined cross-platform data for these 160 genes could predict the prognosis of 102 independent test samples with 71% accuracy. Furthermore it correctly predicted the outcome for 85/102 (83%) NB patients through the leave-one-out cross-validation approach. Our study showed that gene expression studies performed in different platforms could be integrated for prognosis analysis after removing variation resulting from different platforms.     

RefSeq Refinements of UniGene-Based Gene Matching Improve the Correlation of Expression Measurements Between Two Microarray Platforms

Matching genes across microarray platforms is a critical step in meta-analysis. Standard practice uses UniGene to match genes. Numerous studies have found poor correlations between platforms when using UniGene matching.We profiled samples from 33 breast cancer patients on two different microarray platforms (Affymetrix and cDNA) and investigated gene matching. Our results confirmed that UniGene-based matching led to poor correlations of gene expression between platforms. Using RefSeq, a database maintained by the National Center for Biotechnology Information (NCBI), we developed and implemented a new method to refine gene matching. We found that the correlations between gene expression measurements were substantially higher after the RefSeq matching. Our approach differs from previously reported sequence-matching approaches and retains useful expression measurements. It is a sensible approach for matching probes across platforms.We conclude that UniGene alone is insufficient to match genes across platforms. Refined matching based on RefSeq significantly improves the quality of matches.     

Independence and reproducibility across microarray platforms

Microarrays have been widely used for the analysis of gene expression, but the issue of reproducibility across platforms has yet to be fully resolved. To address this apparent problem, we compared gene expression between two microarray platforms: the short oligonucleotide Affymetrix Mouse Genome 430 2.0 GeneChip and a spotted cDNA array using a mouse model of angiotensin II-induced hypertension. RNA extracted from treated mice was analyzed using Affymetrix and cDNA platforms and then by quantitative RT-PCR (qRT-PCR) for validation of specific genes. For the 11,710 genes present on both arrays, we assessed the relative impact of experimental treatment and platform on measured expression and found that biological treatment had a far greater impact on measured expression than did platform for more than 90% of genes, a result validated by qRT-PCR. In the small number of cases in which platforms yielded discrepant results, qRT-PCR generally did not confirm either set of data, suggesting that sequence-specific effects may make expression predictions difficult to make using any technique.