Optimising experimental design for high-throughput phenotyping in mice: a case study

To further the functional annotation of the mammalian genome, the Sanger Mouse Genetics Programme aims to generate and characterise knockout mice in a high-throughput manner. Annually, approximately 200 lines of knockout mice will be characterised using a standardised battery of phenotyping tests covering key disease indications ranging from obesity to sensory acuity. From these findings secondary centres will select putative mutants of interest for more in-depth, confirmatory experiments. Optimising experimental design and data analysis is essential to maximise output using the resources with greatest efficiency, thereby attaining our biological objective of understanding the role of genes in normal development and disease. This study uses the example of the noninvasive blood pressure test to demonstrate how statistical investigation is important for generating meaningful, reliable results and assessing the design for the defined research objectives. The analysis adjusts for the multiple-testing problem by applying the false discovery rate, which controls the number of false calls within those highlighted as significant. A variance analysis finds that the variation between mice dominates this assay. These variance measures were used to examine the interplay between days, readings, and number of mice on power, the ability to detect change. If an experiment is underpowered, we cannot conclude whether failure to detect a biological difference arises from low power or lack of a distinct phenotype, hence the mice are subjected to testing without gain. Consequently, in confirmatory studies, a power analysis along with the 3Rs can provide justification to increase the number of mice used.     

Practical aspects of experimental design in animal research

A brief overview is presented of the key steps involved in designing a research animal experiment, with reference to resources that specifically address each topic of discussion in more detail. After an idea for a research project is conceived, a thorough review of the literature and consultation with experts in that field are pursued to refine the problem statement and to assimilate background information that is necessary for the experimental design phase. A null and an alternate hypothesis that address the problem statement are then formulated, and only then is the specific design of the experiment developed. Likely the most critical step in designing animal experiments is the identification of the most appropriate animal model to address the experimental question being asked. Other practical considerations include defining the necessary control groups, randomly assigning animals to control/treatment groups, determining the number of animals needed per group, evaluating the logistics of the actual performance of the animal experiments, and identifying the most appropriate statistical analyses and potential collaborators experienced in the area of study. All of these factors are critical to designing an experiment that will generate scientifically valid and reproducible data, which should be considered the ultimate goal of any scientific investigation.     

Pedometer and Human Energy Balance Applications for Science Instruction

Teachers can use pedometers to facilitate inquiry learning and show students the need for mathematics in scientific investigation. The authors conducted activities with secondary students that investigated intake and expenditure components of the energy balance algorithm, which led to inquiries about pedometers and related data. By investigating the accuracy of pedometers and variables that may impact reported step counts, students can better understand experimental design and statistical concepts. Students can also examine other data (distance walked, kilocalories expended) using multifunction pedometers and apply the concepts of correlation and regression. This topic fits well with thematic learning and responds to concerns about excess energy intake and insufficient physical activity in the U.S. population.     

Guidelines for the design and statistical analysis of experiments in papers submitted to ATLA

In vitro experiments need to be well designed and correctly analysed if they are to achieve their full potential to replace the use of animals in research. An "experiment" is a procedure for collecting scientific data in order to answer a hypothesis, or to provide material for generating new hypotheses, and differs from a survey because the scientist has control over the treatments that can be applied. Most experiments can be classified into one of a few formal designs, the most common being completely randomised, and randomised block designs. These are quite common with in vitro experiments, which are often replicated in time. Some experiments involve a single independent (treatment) variable, while other "factorial" designs simultaneously vary two or more independent variables, such as drug treatment and cell line. Factorial designs often provide additional information at little extra cost. Experiments need to be carefully planned to avoid bias, be powerful yet simple, provide for a valid statistical analysis and, in some cases, have a wide range of applicability. Virtually all experiments need some sort of statistical analysis in order to take account of biological variation among the experimental subjects. Parametric methods using the t test or analysis of variance are usually more powerful than non-parametric methods, provided the underlying assumptions of normality of the residuals and equal variances are approximately valid. The statistical analyses of data from a completely randomised design, and from a randomised-block design are demonstrated in Appendices 1 and 2, and methods of determining sample size are discussed in Appendix 3. Appendix 4 gives a checklist for authors submitting papers to ATLA.     

Baseline requirements for assessment of mining impact using biological monitoring

Abstract Biological monitoring programmes for environmental protection should provide for both early detection of possible adverse effects, and assessment of the ecological significance of these effects. Monitoring techniques are required that include responses sensitive to the impact, that can be subjected to rigorous statistical analysis and for which statistical power is high. Such issues in baseline research of‘what and how to measure?’and‘for how long?’have been the focus of a programme being developed to monitor and assess effects of mining operations on the essentially pristine, freshwater ecosystems of the Alligator Rivers Region (ARR) in tropical northern Australia. Application of the BACIP (Before, After, Control, Impact, Paired differences) design, utilizing a form of temporal replication, to univariate (single species) and multivariate (community) data is described. The BACIP design incorporates data from single control and impact sites. We argue for modification of the design for particular studies conducted in streams, to incorporate additional independent control sites from adjacent catchments. Inferential power, by way of (i) more confidently attributing cause to an observed change and (ii) providing information about the ecological significance of the change, will be enhanced using a modified BACIP design. In highly valued environments such as the ARR, monitoring programmes require application of statistical tests with high power to guarantee that an impact no greater than a prescribed amount has gone undetected. A minimum number of baseline years using the BACIP approach would therefore be required in order to achieve some desired level of statistical power. We describe the results of power analyses conducted on 2–5 years (depending upon the technique) of baseline data from streams of the ARR and discuss the implications of these results for management.     

A systematic statistical linear modeling approach to oligonucleotide array experiments

We outline and describe steps for a statistically rigorous approach to analyzing probe-level Affymetrix GeneChip data. The approach employs classical linear mixed models and operates on a gene-by-gene basis. Forgoing any attempts at gene presence or absence calls, the method simultaneously considers the data across all chips in an experiment. Primary output includes precise estimates of fold change (some as low as 1.1), their statistical significance, and measures of array and probe variability. The method can accommodate complex experiments involving many kinds of treatments and can test for their effects at the probe level. Furthermore, mismatch probe data can be incorporated in different ways or ignored altogether. Data from an ionizing radiation experiment on human cell lines illustrate the key concepts.     

The application of MANOVA to analyse <Emphasis Type="Italic">Arabidopsis thaliana</Emphasis> metabolomic data from factorially designed experiments

Metabolomic technologies produce complex multivariate datasets and researchers are faced with the daunting task of extracting information from these data. Principal component analysis (PCA) has been widely applied in the field of metabolomics to reduce data dimensionality and for visualising trends within the complex data. Although PCA is very useful, it cannot handle multi-factorial experimental designs and, often, clear trends of biological interest are not observed when plotting various PC combinations. Even if patterns are observed, PCA provides no measure of their significance. Multivariate analysis of variance (MANOVA) applied to these PCs enables the statistical evaluation of main treatments and, more importantly, their interactions within the experimental design. The power and scope of MANOVA is demonstrated through two different factorially designed metabolomic investigations using Arabidopsis ethylene signalling mutants and their wild-type. One investigation has multiple experimental factors including challenge with the economically important pathogen Botrytis cinerea and also replicate experiments, while the second has different sample preparation methods and one level of replication ‘nested’ within the design. In both investigations there are specific factors of biological interest and there are also factors incorporated within the experimental design, which affect the data. The versatility of MANOVA is displayed by using data from two different metabolomic techniques; profiling using direct injection mass spectroscopy (DIMS) and fingerprinting using fourier transform infra-red (FT-IR) spectroscopy. MANOVA found significant main effects and interactions in both experiments, allowing a more complete and comprehensive interpretation of the variation within each investigation, than with PCA alone. Canonical variate analysis (CVA) was applied to investigate these effects and their biological significance. In conclusion, the application of MANOVA followed by CVA provided extra information than PCA alone and proved to be a valuable statistical addition in the overwhelming task of analysing metabolomic data.     

Use of mixture distributions to deconvolute the behavior of "hits" and controls in high-throughput screening data

The stochastic nature of high-throughput screening (HTS) data indicates that information may be gleaned by applying statistical methods to HTS data. A foundation of parametric statistics is the study and elucidation of population distributions, which can be modeled using modern spreadsheet software. The methods and results described here use fundamental concepts of statistical population distributions analyzed using a spreadsheet to provide tools in a developing armamentarium for extracting information from HTS data. Specific examples using two HTS kinase assays are analyzed. The analyses use normal and gamma distributions, which combine to form mixture distributions. HTS data were found to be described well using such mixture distributions, and deconvolution of the mixtures to the constituent gamma and normal parts provided insight into how the assays performed. In particular, the proportion of hits confirmed was predicted from the original HTS data and used to assess screening assay performance. The analyses also provide a method for determining how hit thresholds--values used to separate active from inactive compounds--affect the proportion of compounds verified as active and how the threshold can be chosen to optimize the selection process.     

Model-based analysis on the relationship of signal quality to real-time extraction of information in bioprocesses

Quality by design (QbD) is a current structured approach to design processes yielding a quality product. Knowledge and process understanding cannot be achieved without proper experimental data; hence requirements for measurement error and frequency of measurement of bioprocess variables have to be defined. In this contribution, a model-based approach is used to investigate impact factors on calculated rates to predict the obtainable information from real-time measurements (= signal quality). Measurement error, biological activity, and averaging window (= period of observation) were identified as biggest impact factors on signal quality. Moreover, signal quality has been set in context with a quantifiable measure using statistical error testing, which can be used as a benchmark for process analytics and exploitation of data. Results have been validated with data from an E. coli batch process. This approach is useful to get an idea which process dynamics can be observed with a given bioprocess setup and sampling strategy beforehand.     

Process analytical technology (PAT) for biopharmaceutical products: Part II. Concepts and applications

Implementing real‐time product quality control meets one or both of the key goals outlined in FDA's PAT guidance: “variability is managed by the process” and “product quality attributes can be accurately and reliably predicted over the design space established for materials used, process parameters, manufacturing, environmental, and other conditions.” The first part of the paper presented an overview of PAT concepts and applications in the areas of upstream and downstream processing. In this second part, we present principles and case studies to illustrate implementation of PAT for drug product manufacturing, rapid microbiology, and chemometrics. We further present our thoughts on how PAT will be applied to biotech processes going forward. The role of PAT as an enabling component of the Quality by Design framework is highlighted. Integration of PAT with the principles stated in the ICH Q8, Q9, and Q10 guidance documents is also discussed. Biotechnol. Bioeng. 2010; 105: 285–295. Published 2009 Wiley Periodicals, Inc.     

The design of in vivo multifraction experiments to estimate the alpha-beta ratio

Using statistical methods, the designs of multifraction experiments which are likely to give the most precise estimate of the alpha-beta ratio in the linear-quadratic model are investigated. The aim of the investigation is to try to understand what features of an experimental design make it efficient for estimating alpha/beta rather than to recommend a specific design. A plot of the design on an nd2 versus nd graph is suggested, and this graph is called the design plot. The best designs are those which have a large spread in the isoeffect direction in the design plot, which means that a wide range of doses per fraction should be used. For binary response assays, designs with expected response probabilities near to 0.5 are most efficient. Furthermore, dose points with expected response probabilities outside the range 0.1 to 0.9 contribute negligibly to the efficiency with which alpha/beta can be estimated. For "top-up" experiments, the best designs are those which replace as small a portion as possible of the full experiment with the top-up scheme. In addition, from a statistical viewpoint, it makes no difference whether a single large top-up dose or several smaller top-up doses are used; however, other considerations suggest that two or more top-up doses may be preferable. The practical realities of designing experiments as well as the somewhat idealized statistical considerations are discussed.     

DNA microarray analyses in higher plants

DNA microarrays were originally devised and described as a convenient technology for the global analysis of plant gene expression. Over the past decade, their use has expanded enormously to cover all kingdoms of living organisms. At the same time, the scope of applications of microarrays has increased beyond expression analyses, with plant genomics playing a leadership role in the on-going development of this technology. As the field has matured, the rate-limiting step has moved from that of the technical process of data generation to that of data analysis. We currently face major problems in dealing with the accumulating datasets, not simply with respect to how to archive, access, and process the huge amounts of data that have been and are being produced, but also in determining the relative quality of the different datasets. A major recognized concern is the appropriate use of statistical design in microarray experiments, without which the datasets are rendered useless. A vigorous area of current research involves the development of novel statistical tools specifically for microarray experiments. This article describes, in a necessarily selective manner, the types of platforms currently employed in microarray research and provides an overview of recent activities using these platforms in plant biology.     

A database application for long‐term ecological field experiments

Problem: A series of long‐term field experiments is described, with particular reference to monitoring and quality control. This paper addresses problems in data‐management of particular importance for long‐term studies, including data manipulation, archiving, quality assessment, and flexible retrieval for analysis Method: The problems were addressed using a purpose‐built database system, using commercial software and running under Microsoft Windows. Conclusion: The database system brings many advantages compared to available software, including significantly improved quality checking and access. The query system allows for easy access to data sets thus improving the efficiency of analysis. Quality assessments of the initial dataset demonstrated that the database system can also provide general insight into types and magnitudes of error in data‐sets. Finally, the system can be generalised to include data from a number of different projects, thus simplifying data manipulation for meta‐analysis.     

Analysing Proportions that Arise from Factorial Experiments when the Underlying Proportions are Heterogeneous Within Groups

The analysis of proportions that arise from experiments that involve several factors is considered when there is heterogeneity of the underlying proportions within each combination of the levels of the factors. An analysis is described which can be implemented using a standard statistical program. This also provides an approximate analysis when the data are assumed to be Beta-binomially distributed.     

Functional Mixed-Effect Models for Electrophysiological Responses

In electro/psychophysiological experiments, linear mixed-effect modeling is an effective statistical technique for data repeatedly observed from the same experimental participants or stimulus items. This review describes the application of mixed-effect modeling to functional responses, in particular those observed in event-related EEG or MEG experiments, using a discrete wavelet transform. The technique is illustrated with a design with several covariates, and procedures for generating posterior samples and computing a Bayesian false discovery rate are described. Neirofiziologiya/Neurophysiology, Vol. 41, No. 1, pp. 79–87, January–February, 2009.     

Design and statistical methods in studies using animal models of development

Experiments involving neonates should follow the same basic principles as most other experiments. They should be unbiased, be powerful, have a good range of applicability, not be excessively complex, and be statistically analyzable to show the range of uncertainty in the conclusions. However, investigation of growth and development in neonatal multiparous animals poses special problems associated with the choice of "experimental unit" and differences between litters: the "litter effect." Two main types of experiments are described, with recommendations regarding their design and statistical analysis: First, the "between litter design" is used when females or whole litters are assigned to a treatment group. In this case the litter, rather than the individuals within a litter, is the experimental unit and should be the unit for the statistical analysis. Measurements made on individual neonatal animals need to be combined within each litter. Counting each neonate as a separate observation may lead to incorrect conclusions. The number of observations for each outcome ("n") is based on the number of treated females or whole litters. Where litter sizes vary, it may be necessary to use a weighted statistical analysis because means based on more observations are more reliable than those based on a few observations. Second, the more powerful "within-litter design" is used when neonates can be individually assigned to treatment groups so that individuals within a litter can have different treatments. In this case, the individual neonate is the experimental unit, and "n" is based on the number of individual pups, not on the number of whole litters. However, variation in litter size means that it may be difficult to perform balanced experiments with equal numbers of animals in each treatment group within each litter. This increases the complexity of the statistical analysis. A numerical example using a general linear model analysis of variance is provided in the Appendix. The use of isogenic strains should be considered in neonatal research. These strains are like immortal clones of genetically identical individuals (i.e., they are uniform, stable, and repeatable), and their use should result in more powerful experiments. Inbred females mated to males of a different inbred strain will produce F1 hybrid offspring that will be uniform, vigorous, and genetically identical. Different strains may develop at different rates and respond differently to experimental treatments.     

Design and analysis issues in quantitative proteomics studies

Quantitative proteomics is the comparison of distinct proteomes which enables the identification of protein species which exhibit changes in expression or post-translational state in response to a given stimulus. Many different quantitative techniques are being utilized and generate large datasets. Independent of the technique used, these large datasets need robust data analysis to ensure valid conclusions are drawn from such studies. Approaches to address the problems that arise with large datasets are discussed to give insight into the types of statistical analyses of data appropriate for the various experimental strategies that can be employed by quantitative proteomic studies. This review also highlights the importance of employing a robust experimental design and highlights various issues surrounding the design of experiments. The concepts and examples discussed within will show how robust design and analysis will lead to confident results that will ensure quantitative proteomics delivers.