The importance of the prediction model in the validation of alternative tests

An overview is presented of the validation process adopted by the European Centre for the Validation of Alternative Methods, with particular emphasis on the central role of the prediction model (PM). The development of an adequate PM is considered to be just as important as the development of an adequate test system, since the validity of an alternative test can only be established when both components (the test system and the PM) have successfully undergone validation. It is argued, however, that alternative tests and their associated PMs do not necessarily need to undergo validation at the same time, and that retrospective validation may be appropriate when a test system is found to be reliable, but the case for its relevance remains to be demonstrated. For an alternative test to be considered "scientifically valid", it is necessary for three conditions to be fulfilled, referred to here as the criteria for scientific relevance, predictive relevance, and reliability. A minimal set of criteria for the acceptance of any PM is defined, but it should be noted that required levels of predictive ability need to be established on a case-by-case basis, taking into account the inherent variability of the alternative and in vivo test data. Finally, in view of the growing shift in emphasis from the use of stand-alone alternative tests to alternative testing strategies, the importance of making the PM an integral part of the testing strategy is discussed.     

Peer review of validation studies: an assessment of the role of the OECD by reference to the validation of the uterotrophic assay for endocrine disruptors

The involvement of the OECD in managing the validation of the rat uterotrophic assay for endocrine disruptors, and in organising the peer review of the results of this study, has been assessed and compared with the many conclusions and recommendations in several published reports of international workshops on validation, and information in guidance documents, produced by the European Centre for the Validation of Alternative Methods (ECVAM), the US Interagency Coordinating Committee on the Validation of Alternative Methods (ICCVAM) and the OECD itself. It is concluded that the OECD has not followed the recommendations for full transparency and independence of the peer-review process. This is based on the fact that it has published a draft guidance document that differs from the report of a recent OECD workshop on validation, in such a way as to give the OECD the flexibility to fully control the peer-review process and, in so doing, to avoid full transparency. Comparison of the timing of the organisation of workshops by the OECD and the progression of the validation study, together with the fact that a draft test guideline for the assay was written before completion of the peer review, suggest that the OECD has given a higher priority to the expedition of the validation and regulatory acceptance of the uterotrophic assay than it has to good scientific and logistical practice. This severely undermines its credibility in the validation process, so, in order for the OECD to be rightly perceived as an honest broker, it is recommended that the OECD should play no role in the validation of new or revised tests, until after they have been successfully validated, peer reviewed, and endorsed by the appropriate authorities, and are ready for test guideline development. With regard to the on-going OECD validation studies of other in vivo assays for endocrine disruptors, the OECD should take immediate steps to ensure full independence and transparency of their peer review.     

A modular approach to the ECVAM principles on test validity

The European Centre for the Validation of Alternative Methods (ECVAM) proposes to make the validation process more flexible, while maintaining its high standards. The various aspects of validation are broken down into independent modules, and the information necessary to complete each module is defined. The data required to assess test validity in an independent peer review, not the process, are thus emphasised. Once the information to satisfy all the modules is complete, the test can enter the peer-review process. In this way, the between-laboratory variability and predictive capacity of a test can be assessed independently. Thinking in terms of validity principles will broaden the applicability of the validation process to a variety of tests and procedures, including the generation of new tests, new technologies (for example, genomics, proteomics), computer-based models (for example, quantitative structure-activity relationship models), and expert systems. This proposal also aims to take into account existing information, defining this as retrospective validation, in contrast to a prospective validation study, which has been the predominant approach to date. This will permit the assessment of test validity by completing the missing information via the relevant validation procedure: prospective validation, retrospective validation, catch-up validation, or a combination of these procedures.     

ECVAM-ICCVAM: prospects for future collaboration

The level and complexity of testing for hazard and risk assessment of marketed products and environmental agents has increased substantially over time, resulting in the use of greater numbers of both animals and humans for testing. Today, industry and regulatory bodies worldwide face increasing pressures to demonstrate responsible utilisation of laboratory animals, to limit their use, and to employ alternative non-animal tests. Institutions have also been established to identify, encourage development of, conduct research on, and validate new, improved, and surrogate test methods that will reduce and replace animal use. Two such organisations are ECVAM and the Interagency Coordinating Committee for the Validation of Alternative Methods (ICCVAM). As the evolutionary changes occurring in the field of toxicology result in an unprecedented increase in the introduction of alternative methodologies, these will strain the capacities of such alternative methods institutions. That realisation is causing a shift in thinking and creating an impetus to seek approaches by which to collaborate and develop more-efficient operational procedures for the validation and regulatory acceptance of alternative methods. Similarities in objectives, functions, scientific standards, and commitment to the principles of validation and animal welfare support the value of a cooperative arrangement between ECVAM and ICCVAM, to minimise duplication of effort, maximise productivity, and influence the international adoption of alternative tests. Opportunities for ECVAM-ICCVAM collaboration are discussed, which illustrate the feasibility and potential benefits of such a partnership.     

The ECVAM workshops: a critical assessment of their impact on the development,validation and acceptance of alternative methods

ECVAM initiated its workshop programme in 1994, to enable it to become well informed about the state of the art of non-animal test development and validation, and about the possible incorporation of alternatives into regulatory requirements for safety testing. Fifty-one such workshops have been held on specific topics, up to 2002. In these workshops, the current status of in vitro tests and their potential uses were reviewed and recommendations were made as to the best ways forward to progress and enhance the utilisation of in vitro methods. Reports for 46 of these workshops have been published in ATLA. Most of the workshops focused on in vitro replacement methods, although an increasing number have dealt with reduction and refinement. The recommendations in the ECVAM workshops have been progressed further by: a) the formation of ECVAM task forces; b) the organisation of further workshops; c) the activities of scientific committees; d) the provision of earmarked research funding; and e) the conduct of validation studies. Examples of each of these activities are discussed. Some individual workshops are covered in more detail and several recommendations that have so far not been acted on are also considered. The workshops and their reports have had a substantial effect on the development and implementation of alternative methods, and have been a major factor in contributing to the success of the first nine years of ECVAM's existence. It is strongly recommended that ECVAM continues to organise workshops and to publish their findings, and several suggestions are made for topics of future workshops.     

Product-specific validation of a serological potency test for release of Leptospira vaccines in the European Union

Historically in the European Union, all Leptospira vaccines were released using the European Pharmacopoeia (Ph. Eur.) hamster potency assay. Recently, there has been a shift toward alternatives that offer either refinement of testing or replacement of animals for product release. This is being driven by animal welfare concerns but also by a drive to have more consistent, cheaper, and faster batch release tests. This publication discusses one such example of a multicomponent canine vaccine that includes three Leptospira serovars and has recently been registered in the European Union. The potency release test is a refinement because it uses rabbit serology rather than hamster challenge. This publication covers the principles of the test method, challenges faced during its development and registration, and discussion about benefits and limitations of this method. It concludes with a view of how the use of serology testing could fit into an overall strategy to move to fully in vitro testing by adopting a consistency approach.     

Biostatistical methods for the validation of alternative methods for in vitro toxicity testing

Statistical methods for the validation of toxicological in vitro test assays are developed and applied. Validation is performed either in comparison with in vivo assays or in comparison with other in vitro assays of established validity. Biostatistical methods are presented which are of potential use and benefit for the validation of alternative methods for the risk assessment of chemicals, providing at least an equivalent level of protection through in vitro toxicity testing to that obtained through the use of current in vivo methods. Characteristic indices are developed and determined. Qualitative outcomes are characterised by the rates of false-positive and false-negative predictions, sensitivity and specificity, and predictive values. Quantitative outcomes are characterised by regression coefficients derived from predictive models. The receiver operating characteristics (ROC) technique, applicable when a continuum of cut-off values is considered, is discussed in detail, in relation to its use for statistical modelling and statistical inference. The methods presented are examined for their use for the proof of safety and for toxicity detection and testing. We emphasise that the final validation of toxicity testing is human toxicity, and that the in vivo test itself is only a predictor with an inherent uncertainty. Therefore, the validation of the in vitro test has to account for the vagueness and uncertainty of the "gold standard" in vivo test. We address model selection and model validation, and a four-step scheme is proposed for the conduct of validation studies. Gaps and research needs are formulated to improve the validation of alternative methods for in vitro toxicity testing.     

Report of the EPAA-ECVAM workshop on the validation of Integrated Testing Strategies (ITS)

The use of Integrated Testing Strategies (ITS) permits the combination of diverse types of chemical and toxicological data for the purposes of hazard identification and characterisation. In November 2008, the European Partnership for Alternative Approaches to Animal Testing (EPAA), together with the European Centre for the Validation of Alternative Methods (ECVAM), held a workshop on Overcoming Barriers to Validation of Non-animal Partial Replacement Methods/Integrated Testing Strategies, in Ispra, Italy, to discuss the extent to which current ECVAM approaches to validation can be used to evaluate partial replacement in vitro test methods (i.e. as potential ITS components) and ITS themselves. The main conclusions of these discussions were that formal validation was only considered necessary for regulatory purposes (e.g. the replacement of a test guideline), and that current ECVAM approaches to validation should be adapted to accommodate such test methods. With these conclusions in mind, a follow-up EPAA-ECVAM workshop was held in October 2009, to discuss the extent to which existing validation principles are applicable to the validation of ITS test methods, and to develop a draft approach for the validation of such test methods and/or overall ITS for regulatory purposes. This report summarises the workshop discussions that started with a review of the current validation methodologies and the presentation of two case studies (skin sensitisation and acute toxicity), before covering the definition of ITS and their components, including their validation and regulatory acceptance. The following main conclusions/recommendations were made: that the validation of a partial replacement test method (for application as part of a testing strategy) should be differentiated from the validation of an in vitro test method for application as a stand-alone replacement, especially with regard to its predictive capacity; that, in the former case, the predictive capacity of the whole testing strategy (rather than of the individual test methods) would be more important, especially if the individual test methods had a high biological relevance; that ITS allowing for flexible and ad hoc approaches cannot be validated, whereas the validation of clearly defined ITS would be feasible, although practically quite difficult; and that test method developers should be encouraged to develop and submit to ECVAM not only full replacement test methods, but also partial replacement methods to be placed as parts of testing strategies. The added value from the formal validation of testing strategies, and the requirements needed in view of regulatory acceptance of the data, require further informed discussion within the EPAA forum on the basis of case studies provided by industry.     

The use of bootstrap resampling to assess the variability of Draize tissue scores

The acute dermal and ocular effects of chemicals are generally assessed by performing the Draize skin and eye tests, respectively. Because the animal data obtained in these tests are also used for the development and validation of alternative methods for skin and eye irritation, it is important to assess the inherent variability of the animal data, since this variability places an upper limit on the predictive performance that can be expected of any alternative model. The statistical method of bootstrap resampling was used to estimate the variability arising from the use of different animals and time-points, and the estimates of variability were used to determine the maximal extent to which Draize test tissue scores can be predicted.     

Assessment of the eye irritating properties of chemicals by applying alternatives to the Draize rabbit eye test: the use of QSARs and in vitro tests for the classification of eye irritation

Huggins has reported on the current situation relating to the development of alternatives to the Draize eye irritation test with rabbits, and an ECVAM Working Group have reviewed the efforts needed in order to replace this animal test within the next 10 years by using the results of non-animal assessment methods. Our report reviews regulatory experience gained over the last 20 years with the EU chemicals notification procedure with respect to the assessment of eye lesions observed in Draize tests. The nature of eye lesions and their importance for classification and labelling of possible hazards to human eyes are evaluated and discussed, with a view to promoting the development of specific in vitro assays which are able to discriminate between eye damage, moderate eye irritation, and minor irritation effects which are completely reversible within a few days. Structural alerts for the prediction of eye irritation/corrosion hazards to be classified and labelled according to international classification criteria, are presented, which should be validated in accordance with internationally agreed (OECD) principles for (Q)SAR system validation. Physicochemical limit values for prediction of the absence of any eye irritation potential relevant for human health can make available a definition of the applicability domains of alternative methods developed for the replacement of the Draize eye irritation test.     

Ethological validation and the assessment of anxiety-like behaviours: methodological comparison of classical analyses and structural approaches

The research on emotional reactivity usually implies the use of standardised behavioural tests that provide a quick idea of the effect of a treatment on the reactivity of subjects to potentially dangerous situations. Many validity criteria have been considered to evaluate these tests. This validity concept supports the idea that animals' behaviour in these tests model human anxiety. Generally, those criteria repeatedly labelled as "ethological validation" refer to the analogy between animals and human in the meaning of the test situation. Although the content of the ethological validation concept is heterogeneous, it is steadily related to a fixed interpretation of the behavioural items produced in a given experimental setting. The basic assumption of such reasoning is that the behavioural items would always be expressed in the same behavioural context whatever the subject, its gender, strain or species, thoroughly asserting a predefined subjective state. Using multivariate and textual analysis, we found evidence that the "ethological validation" recourse to an a priori interpretation for a given behavioural variable may be deceptive. We defend the idea that the meaning of a behavioural variable should be restricted to the general context where it arose. Theoretical propositions and methodological options are discussed.     

Targeted proteomics in biomarker validation: detection and quantification of proteins using a multi-dimensional peptide separation strategy

Authentic biomarkers, distilling the essence of a complex, functionally significant process in a mammalian system into a precise, physicochemical measurement have been implicated as a tool of increasing importance for drug discovery and development. However, even in spite of recent technological advances, validating a new biomarker candidate, where generation of suitable antibodies is required, is still a long-lasting task. Methods to accelerate initial validation by MS approaches have been suggested, but all methods described so far are associated with serious drawbacks, finally leading to non-generic methods of detection and quantification. Moreover, when complex body fluids are used as samples, efficient debulking strategies are crucial to open a window of analytical sensitivity in the ng/mL range, where many diagnostically relevant analytes are present. Here we report the proof-of-principle of a multi-dimensional strategy for accelerated initial validation of biomarker candidates by MS, which promises to be generally applicable, sensitive and quantitative. The method presented employs a combination of electrophoretic and chromatographic steps on the peptide level, followed by MS quantification using isotopically labeled synthetic peptides as internal standards. Our proposed workflow includes up to four dimensions, finally resulting in a desired LOD sufficient to detect and quantify diagnostically relevant analytes from complex samples. Although the current state of the method only represents a starting point for further validation and development, it reveals great potential in biomarker validation.     

Protein biomarker discovery and validation: the long and uncertain path to clinical utility

Better biomarkers are urgently needed to improve diagnosis, guide molecularly targeted therapy and monitor activity and therapeutic response across a wide spectrum of disease. Proteomics methods based on mass spectrometry hold special promise for the discovery of novel biomarkers that might form the foundation for new clinical blood tests, but to date their contribution to the diagnostic armamentarium has been disappointing. This is due in part to the lack of a coherent pipeline connecting marker discovery with well-established methods for validation. Advances in methods and technology now enable construction of a comprehensive biomarker pipeline from six essential process components: candidate discovery, qualification, verification, research assay optimization, biomarker validation and commercialization. Better understanding of the overall process of biomarker discovery and validation and of the challenges and strategies inherent in each phase should improve experimental study design, in turn increasing the efficiency of biomarker development and facilitating the delivery and deployment of novel clinical tests.     

化妆品眼刺激试验替代方法标准化与展望

眼刺激试验是化妆品安全评价的主要测试项目,目前正尝试建立代替传统动物实验的体外方法,这些方法处于研发、验证和认可的不同阶段。尽管还没有单一体外试验能完全替代兔眼实验,但系统分析替代方法建立的原理、科学相关性和标准化程度的不同,有助于合理选择运用体外方法指导化妆品眼刺激的标识分类、质量检验、产品开发和机制研究。     

Comparison and validation of novel pyrogen tests based on the human fever reaction

The Limulus amoebocyte lysate (LAL) test has replaced about 80% of the use of the rabbit pyrogen test. Ideally, human-based in vitro tests are needed, to replace the remaining use of the rabbit test and the use of the LAL test. The progress of an EU-funded project is described, in which a number of in vitro tests, based on the human fever reaction are passing through a prevalidation study on the way to evaluation in a formal validation study.     

Designing validation studies more efficiently according to the modular approach: retrospective analysis of the EPISKIN test for skin corrosion

It is claimed that the modular approach to validation, which involves seven independent modules, will make the assessment of test validity more flexible and more efficient. In particular, the aspects of between-laboratory variability and predictive capacity are formally separated. Here, the main advantage of the approach is to offer the opportunity for reduced labour, and thus to allow study designs to be more time efficient and cost effective. The impact of this separation was analysed by taking the ECVAM validation study on in vitro methods for skin corrosivity as an example of a successful validation study - two of its methods triggered new OECD test guidelines. Lean study designs, which reduced the number of tests required by up to 60%, were simulated with the original validation data for the EPISKIN model. By using resampling techniques, we were able to demonstrate the effects of the lean designs on three between-laboratory variability measures and on the predictive capacity in terms of sensitivity and specificity, in comparison with the original study. Overall, the study results, especially the levels of confidence, were only slightly affected by the lean designs that were modelled. It is concluded that the separation of the two modules is a promising way to speed-up prospective validation studies and to substantially reduce costs, without compromising study quality.     

Estimating biofilm reaction kinetics using hybrid mechanistic-neural network rate function model

This work describes an alternative method for estimation of reaction rate of a biofilm process without using a model equation. A first principles model of the biofilm process is integrated with artificial neural networks to derive a hybrid mechanistic-neural network rate function model (HMNNRFM), and this combined model structure is used to estimate the complex kinetics of the biofilm process as a consequence of the validation of its steady state solution. The performance of the proposed methodology is studied with the aid of the experimental data of an anaerobic fixed bed biofilm reactor. The statistical significance of the method is also analyzed by means of the coefficient of determination (R²) and model efficiency (ME). The results demonstrate the effectiveness of HMNNRFM for estimating the complex kinetics of the biofilm process involved in the treatment of industry wastewater.     

Development and preliminary validation of an antibody filtration-assisted single-dilution chemiluminometric immunoassay for potency testing of Piscirickettsia salmonis vaccines

Challenge with live pathogens could be substituted by serology for many veterinary diseases, however little progress has been made in the development of alternative batch vaccine potency tests for fish. This study reports the development and preliminary validation of a single-dilution filtration-assisted chemiluminometric immunoassay (SD FAL-ELISA) applied to measure anti Piscirickettsia salmonis IgM in individual or pooled serum and mucus samples. The assay was set up to test a single-dilution of the sample. Serum SD FAL-ELISA yielded a sensitivity of 90% and a specificity of 96%. SD FAL-ELISA was applied to evaluate pooled and individual samples from P. salmonis challenge assessments. Relative-light units values (RLU) obtained by SD FAL-ELISA were proportional to antibody levels in serum. RLU values obtained from pooled and individual serum samples increased with the observed relative percent survival (RPS) values, indicating a correlation between protection and specific IgM levels. Results obtained for specific IgM in mucus samples was not related to the RPS, but discriminated the vaccine that yielded high RPS (86.4%) from the others (40.9 and 54.5%). This is the first report on the development of an indirect high-throughput serological assessment for P. salmonis vaccine potency testing using both pooled or individual serum and cutaneous mucus samples.     

The red blood cell phototoxicity test (photohaemolysis and haemoglobin oxidation): EU/COLIPA validation programme on phototoxicity (phase II)

In the EU/COLIPA validation programme on "Photoirritation in vitro", two core tests and a number of mechanistically based tests were carried out to examine their suitability as regulatory tests for phototoxicity testing. In the meantime, one core test, the 3T3 neutral red uptake phototoxicity test (NRU PT) has been validated and has been accepted by ECVAM and the European Commission. The second core test, the red blood cell phototoxicity test (Photo-RBC test), has passed through a prevalidation process during this programme. This test protocol combines two endpoints, photohaemolysis and met-haemoglobin (met-Hb) formation. These endpoints are determined by measuring changes in the optical density of the haemoglobin spectrum at 525 nm and 630 nm, respectively. In addition, a prediction model was inserted into the Standard Operating Procedure (SOP) with two cut-off values: a photohaemolysis factor (PHF) > or = 3.0 for photohaemolysis, and a deltaOD(max) > or = 0.05 for met-Hb formation. Three laboratories agreed to implement the SOP and to perform the study by testing 30 selected test chemicals (25 phototoxicants and 5 non- phototoxic chemicals). The outcome of the study presents a good overall fit, including acceptable accuracy, sensitivity, and positive predictivity. The specificity and the negative predictivity are comparably low, due to the low number of non-phototoxic substances among the test chemicals. Further analysis of the data showed that the transfer of the SOP from between laboratories could have been more efficient. The results, especially of the lead laboratory, clearly indicate that an experienced laboratory can handle the SOP with high predictivity for phototoxicants and non-phototoxic substances. Finally, it was concluded that the combined Photo-RBC test can be considered as a second in vitro test, which can be used advantageously to obtain some mechanistic information, in particular on photodynamic effects on cellular proteins and biomembranes.     

ECVAM,ECETOC and the EU chemicals policy

ECVAM's initiatives in validation have received significant support from the European Centre for Ecotoxicology and Toxicology of Chemicals (ECETOC), especially through the provision of reference chemical data banks, which contain peer-reviewed, high-quality in vivo data on commercially available chemical substances. Chemicals have been selected from these ECETOC data banks for validation studies on alternative methods for skin corrosion and irritation and for eye irritation and, in addition, an ECETOC task force peer-reviewed the selection and classification, on the basis of in vivo data, of chemicals used in the validation of three alternative methods for developmental toxicity. More recently, ECVAM and ECETOC have been pursuing parallel initiatives on the proposed new EU chemicals policy, with the common goals of ensuring that industry and European Commission resources are used to investigate only those chemicals that pose a significant risk to human health and the environment, and that the Policy requires that any testing which is required follows the Three Rs principles of reduction, refinement and replacement.