Interlaboratory quality control in gynecologic cytopathology using the novel CONQUISTADOR software. Interobserver reproducibility in the Latin American screening study

OBJECTIVE: The recently developed software (CONQUISTADOR), capable of computing all intralaboratory and interlaboratory quality control (QC) indicators, was used to evaluate the diagnostic agreement among 4 cytology laboratories participating in the LAMS Study. STUDY DESIGN: The study was an interlaboratory exchange of specially designed 5 slide sets, each comprising 20 (conventional cytology) slides. At the first step, 80 slides (with "clear-cut" cases) were divided into four sets (A, B, C, D) of 20 specimens, each including inadequate and negative cases as well as in different proportions of all diagnostic TBS 2001 categories. In the second round, a fifth set (E) of 20 slides ("difficult cases") was designed, with all diagnostic categories, ASC and AGC included. Common measures of reproducibility (kappa and weighted kappa), accuracy (SE, SP, PPV, NPV) and 3 indices of diagnostic variability were calculated for sets A-D and set E, separately. RESULTS: For the 5 slide sets together, the weighted kappa was 0.8 (95% CI 0.76-0.85), which is the lower limit of the "almost perfect" ranking of kappa statistics, indicating an excellent interlaboratory agreement. The interlaboratory reproducibility was lower only for the difficult set (E). Similarly, the sensitivity for set E (70.0%) was lower than that (92.1%) for sets A-D. The diagnostic variability indices were not substantially different between the difficult (set E) and clearcut (sets A-D) cases. CONCLUSION: High interlaboratory reproducibility was obtained for sets A-D ("clear-cut" cases), while more interlaboratory variation was evident in the difficult samples. The new CONQUISTADOR software is a valuable tool in calculating the indicators needed in this intralaboratory and interlaboratory.     

Harmonizing lipidomics: NIST interlaboratory comparison exercise for lipidomics using SRM 1950–Metabolites in Frozen Human Plasma

As the lipidomics field continues to advance, self-evaluation within the community is critical. Here, we performed an interlaboratory comparison exercise for lipidomics using Standard Reference Material (SRM) 1950–Metabolites in Frozen Human Plasma, a commercially available reference material. The interlaboratory study comprised 31 diverse laboratories, with each laboratory using a different lipidomics workflow. A total of 1,527 unique lipids were measured across all laboratories and consensus location estimates and associated uncertainties were determined for 339 of these lipids measured at the sum composition level by five or more participating laboratories. These evaluated lipids detected in SRM 1950 serve as community-wide benchmarks for intra- and interlaboratory quality control and method validation. These analyses were performed using nonstandardized laboratory-independent workflows. The consensus locations were also compared with a previous examination of SRM 1950 by the LIPID MAPS consortium. While the central theme of the interlaboratory study was to provide values to help harmonize lipids, lipid mediators, and precursor measurements across the community, it was also initiated to stimulate a discussion regarding areas in need of improvement.     

Robust estimation of multivariate covariance components

In many settings, such as interlaboratory testing, small area estimation in sample surveys, and heritability studies, investigators are interested in estimating covariance components for multivariate measurements. However, the presence of outliers can seriously distort estimates obtained using standard procedures such as maximum likelihood. We propose a procedure based on M-estimation for robustly estimating multivariate covariance components in the presence of outliers; the procedure applies to balanced and unbalanced data. We present an algorithm for computing the robust estimates and examine the performance of the estimator through a simulation study. The estimator is used to find covariance components and identify outliers in a study of variability of egg length and breadth measurements of American coots.     

A critique of the collaborative cytogenetics study to measure and minimize interlaboratory variation.

A statistical reanalysis was performed on the data fecently reported on a 6-laboratory, collaborative cytogenetic study to measure and minimize interlaboratory variation. Three of the laboratories had mean values significantly different from the others on most of the 6 indexes of chemically-induced aberration; one laboratory with values higher and two with values lower. Furthermore, relative variability of the values around the means was consistently lower in one of the 6 participating laborabories. The results of the reanalysis of this collaborative study demonstrates that significant interlaboratory differences exist and that these should be adjusted or diminished before rat cytogenetic analysis can be an effective test system for evaluation of a compound for mutagenic potential.     

Interlaboratory evaluation of VITEK2 system and Sensititre YeastOne® for antifungal susceptibility testing of yeasts isolated from blood cultures against four antifungal agents

An interlaboratory evaluation (seven centers) of VITEK2 System and Sensititre YeastOne? was conducted to test the antifungal susceptibilities of yeasts. The MICs of amphotericin B, fluconazole, flucytosine, and voriconazole were determined for 70 isolates of Candida spp. Our results demonstrated a higher interlaboratory agreement of VITEK 2 System than Sensititre YeastOne?. A good concordance between the two methods was observed for amphotericin B, fluconazole, voriconazole and 5-fluorocytosine (from 81.4% to 88.6%). The study suggests the potential value of the VITEK2 System as a convenient alternative method for testing the susceptibility of yeasts. It also indicates the need for further optimization of MIC endpoint criteria to improve interlaboratory agreement.     

New statistical software for intralaboratory and interlaboratory quality control in clinical cytology. Validation in a simulation study on clinical samples

OBJECTIVE: To design a statistical software package to provide automated calculations of normal and weighted and 3 indices. STUDY DESIGN: Prompted by the lack of commonly available software to compute weighted kappa and the nonproportionate workload needed to calculate our 3 variability indices manually, the new statistical software package was designed. To demonstrate the performance of the new CONQUISTADOR software, a simulation study (both intralaboratory and interlaboratory) was designed using 5,000 clinical samples randomly selected from a data file of > or = 200,000 conventional Pap smears and programmed to become "analyzed" by 12 cytologists in 5 imaginary laboratories. RESULTS: A representative set of both complete and partial outputs provided by the software, in Excel format (Microsoft, Redmond, Washington, U.S.A.) are shown to illustrate the different functions of the program. In the interlaboratory mode, the software calculates accuracy indicators (sensitivity, specificity, positive and negative predictive value, and their 95% CI), which are not common features of regular statistical packages; kappa and weighted kappa; and their 95% CI (comparison of single laboratories to all laboratories and pairwise comparisons between single laboratories). The 3 diagnostic variability indices can be computed separately for all samples or for only the positive samples. In the intralaboratory mode, the software calculates the same indices for individual cytologists. CONCLUSION: The CONQUISTADOR statistical package has properties that are useful in monitoring cytologic laboratory quality in both intralaboratory and interlaboratory settings. The software will be distributed by the National Institute of Health, Rome, for the delivery costs only.     

Statistics of interlaboratory in vitro toxicological studies

Interlaboratory studies are common in toxicology, particularly for the introduction of alternative assays. Numerous papers are available on the statistical analysis of interlaboratory studies, but these deal primarily with the case of a replicated single sample studied in several laboratories. This approach can be used for some assays, but for the majority, the results will be unsatisfactory, i.e. involving great variability between both the dose groups and the laboratories. However, the primary objective of toxicological assays is to achieve similarity between the sizes of effects, rather than to determine absolute values. In the parametric model, the sizes of effects are the studentised differences from the negative control or, for the commonly used dose-response designs, the similarity of the slopes of the dose-response curves. Standard approaches for the estimation of intralaboratory and interlaboratory variability, including Mandel plots, are introduced, and new approaches are presented for demonstrating similarity of effect sizes, with or without assuming a dose-response model. One approach is based on a modification of the parallel-line assay, the other is based on a modification of the interaction contrasts of the analysis of variance. SAS programs are given for all approaches, and real data from an interlaboratory immunotoxicological study are analysed as a demonstration.     

Analysis of heterocyclic amines in food products: interlaboratory studies

A feasibility study and two interlaboratory exercises on the determination of selected heterocyclic amines (HAs) in beef extract, organised in the framework of a European project, are presented. The aim of these exercises was to improve the quality of the laboratories and to evaluate the performance of a standardised analytical method and also the methods currently used by each of the participants for the analysis of these compounds. Three lyophilised portions of a commercial beef material previously spiked with HAs at different concentration levels ranging from 10 to 75 ng g(-1) were used as laboratory reference materials (lot A, B and C). Firstly, a feasibility study was carried out using a test standard solution and the beef extract (lot A), which contained only five HAs. Then, two interlaboratory exercises were carried out using the laboratory reference materials lot B and lot C, containing 10 selected HAs at two different concentration levels, 75 and 10 ng/g, respectively. The results obtained by all participant laboratories using the proposed method showed satisfactory agreement and the CV(%) between-laboratories obtained were from 8.3 to 24.1% for lot B and from 8.7 to 44.5% for lot C. The standardised method evaluated in these collaborative studies is therefore proposed for the analysis of HAs in food material. Moreover, LC-MS is recommended as the most suitable technique for the analysis of a large number of HAs in food samples.     

Enumeration of CD4(+) T-cells in the peripheral blood of HIV-infected patients: an interlaboratory study of the FACSCount system.

The aim of the present study was to assess the interlaboratory reproducibility of the FACSCount system for the enumeration of peripheral blood (PB) CD4(+) T-cells. In each of the seven participating centers, both previously stained and unstained PB samples (n = 49) were received and either analyzed or stained and then analyzed. Interlaboratory reproducibility was checked in two different groups of centers (n = 3 and n = 4) where the study was performed in parallel. In addition, both the intralaboratory precision and accuracy of this system were analyzed in comparison with results obtained with conventional flow cytometry. Accordingly, upon comparing both methods, a high degree of correlation was observed in the total number of CD3(+) T-cells (coefficient of correlation of 0.9750 +/- 0.0184, slope of the best linear fit: 0. 9214 +/- 0.0311, y-intercept of 12 +/- 47) as well as in the number of CD3(+)/CD4(+) (coefficient of correlation of 0.9794 +/- 0.1457, slope of the best linear fit: 0.9463 +/- 0.0753, y-intercept of -11 +/- 36) and CD3(+)/CD8(+) (coefficient of correlation of 0.9728 +/- 0.0192, slope of the best linear fit: 0.9682 +/- 0.0735, y-intercept of 7 +/- 95) major subsets. In addition, low coefficients of variation (CV) were obtained for replicates, indicating the method's high degree of accuracy. The present study shows that with respect to the interlaboratory reproducibility reported for most techniques used for the enumeration of PB CD4(+) T-cells, the FACSCount system results in data with much lower coefficients of variance (CVs) (mean CV of less than 10%). Upon measuring the impact on results of different variables associated with either sample preparation or data acquisition and analysis, our study clearly shows that data acquisition and analysis does not influence the results by increasing variability since the coefficients of variation obtained for samples prepared in the same laboratory under the same conditions and read in different laboratories with different instruments were identical to those obtained for the replicates of the same samples read in each individual center. In contrast, interlaboratory variability, although low, significantly increased when sample preparation was carried out in different laboratories, suggesting that pipetting still represents the major source of variability in the FACSCount system.     

Quality control in haematology: Report of interlaboratory trials in britain

An interlaboratory quality control scheme has been established in Britain by the British Committee for Standards in Haematology. In the first instance this has been confined to haemoglobin, red blood cell count, and packed cell volume. The materials which have been circulated include whole blood, stabilized red cell preparations, lysates, cyanmethaemoglobin solutions, and cyanmethaemoglobin reference preparations. The first two trials have been completed, and there are implications for instrument calibration, dilution techniques, and the use of standards. Periodic interlaboratory trials, at a national level, in conjunction with regular individual intralaboratory quality control procedures, are necessary in order to achieve acceptable levels of accuracy and precision.     

Expert examination of the immunogenic activity and toxicity of the pertussis component of adsorbed and non-adsorbed diphtheria-tetanus-pertussis vaccines. Results of interlaboratory investigations under the WHO project "Reactogenicity and Toxicity of DPT Vaccines"

Since the DPT vaccine is broadly used for the prevention of diphtheria, tetanus and pertussis' its preparations should meet special quality requirements. The present study was aimed at evaluating the reproducibility of laboratory methods utilized to assess the protective activity and toxicity of the pertussis component as well as at examining the feasibility of expert estimations of product quality to enhance the validity of findings. The results of interlaboratory comparative examination of quality parameters of the tested preparations revealed that the routine laboratory quality control methods were not sufficiently standardized as their application in different laboratories did not always produce identical results. The WHO criteria to evaluate the toxicity of pertussis vaccines are far from perfect since vaccines with pronounced toxicity can only be distinguished from those with moderate to mild toxicity after the administration to tested mice of vaccines at a single infant dose of 0.5 ml. To enhance the reproducibility of methods employed in the laboratory, appropriate standard specimens of the preparation should be used serving as a measure of different quality parameters; all test conditions should be also standardized as far as possible. To enable objective quality evaluation of medical biological preparations, the degree of experiment reproducibility should be regularly verified in interlaboratory tests on an international scale as well as inside those countries which have several manufacturers of a given preparation. It appears expedient to set up an international system of expert evaluation of quality of biological preparations by appointing several regional and national centres which meet the requirements for expert laboratories.     

An interlaboratory study of an EPA/Ames/Salmonella test protocol

7 laboratories participated in a collaborative study to evaluate an EPA standard protocol for the Ames test. The study utilized Salmonella typhimurium (strains TA98 and TA100) and 3 metabolic activation levels (0%, 2%, and 10% S9 in the S9 mix). 6 pure chemicals and 2 complex mixtures were tested as coded unknowns. Ability to obtain qualitative results in agreement with published data was less (% agreement) than that reported in an earlier study (% agreement) by de Serres and Ashby (1981) in which each laboratory used its own protocol. The conclusion from analysis of the quantitative data from this interlaboratory Ames study was that both intralaboratory and interlaboratory variations were substantial. Results for the same substance varied by an order of magnitude or more (CV of 115%) when the mutagenic response was measured as the slope of the dose response in revertants/microgram. Taking interlaboratory variation into account, one chemical must be more than an order of magnitude more mutagenic than another (ratio of slopes greater than 10) to have only an even chance of finding a statistically significant difference between the two chemicals at the 5% level. Such large variations must be taken into account when evaluating Ames/Salmonella data.     

Estimating bird species richness: How should repeat surveys be organized in time?

Abstract Estimates of species richness for a given area require that repeat surveys be taken, so that the statistical robustness of the estimate can be assessed. But how should these repeat surveys be organized in time? Here we present a case study of Australian woodland birds, surveyed using the ‘active timed area search’ method, which has become the standard unit for the Australian Bird Atlas, a continental‐scale bird survey. To date, there has been no assessment of how estimates of species richness derived from this method are affected by the temporal organization of the repeat surveys. For instance, can conducting the repeat surveys in sequence on the same day effectively capture richness, or will additional species be obtained by repeating the surveys on different days within a season? If so, does the spacing of the repeat visits throughout the season have an effect? To answer these questions, we surveyed woodland birds in the Mount Lofty Ranges, South Australia, during late spring–summer 1999–2000, and compared the performance of two different temporal configurations of repeat visits to sites: (i) six repeat surveys performed on the same day; and (ii) three repeat surveys on different days. For both, we calculated the average number of species actually sighted and also estimated total species richness. The data supported our hypothesis that the same‐day surveys would yield fewer species and underestimate total species richness. The different‐day repeats captured significantly more species per unit of survey effort, and yielded a higher richness estimate. However, the timespan over which different‐day surveys were conducted within a season did not have a significant influence on species richness estimates, evincing a qualitative advantage to surveying on different days, regardless of the spacing of repeat visits. These results may be of assistance to conservation managers when planning cost‐efficient monitoring regimes.     

Interlaboratory reproducibility of female genital tract cytokine measurements by Luminex: Implications for microbicide safety studies

The interlaboratory reproducibility of cytokine measurements from cervicovaginal samples by Luminex has not been reported. Using cervicovaginal lavage specimens collected on three study days from 12 women participating in a Phase I microbicide study, we measured a panel of eight cytokines in three independent laboratories. Four (IFN-γ, IL-10, IL-17, and TNF) were below the limit of detection in the majority (85%) of samples in either two or all three laboratories, an observation that may guide analyte selection for future studies. Good interlaboratory agreement (intraclass correlation coefficient, r > 0.7) in absolute levels was observed for IL-1β, IL-6, and IL-8, while poor agreement was seen for IFN-α2 (r = 0.47). When considering within-subject change from baseline (pre-product, at study-day 0) to either post-product visit (study-days 7 and 14), IL-1β and IL-6 exhibited good interlaboratory agreement (r > 0.7), while IFN-α2 and IL-8 did not. Future studies addressing the clinical utility of specific biomarkers of inflammation for microbicide trials should consider reproducibility in the context of defining biologically meaningful thresholds of change for candidate biomarkers, ensuring that such change can be reliably distinguished from background variability.     

Study of the intra- and interlaboratory reproducibility of partial single base C-sequencing of the 16S rRNA gene and its applicability for the identification of members of the genus Streptococcus

The use of Single Base C-Sequencing of the first 500 bases of the 16S rRNA-gene (SBCS) combined with capillary electrophoresis was evaluated for the identification of reference strains of 30 different species within the genus Streptococcus. For SBCS, only dd-CTP's are used in the sequencing reactions instead of the four dideoxy bases and the primer is fluorescently labeled. The reproducibility, interlaboratory exchangeability and discriminative power of this method were studied by comparing the patterns obtained in three laboratories under highly standardized conditions. The interlaboratory reproducibility proved to be high, enabling the construction of a common database for the identification of strains belonging to the streptococcal species studied. Most of the examined species generated distinguishable profiles. SBCS did not differentiate between the closely related species S. constellatus and S. intermedius. Also S. thermophilus and S. vestibularis as well as S. mitis and S. pneumoniae showed highly resembling profiles. The previously reported heterogeneity within the species S. equinus was reflected by SBCS. For all other species, strains belonging to the same species generated indistinguishable patterns. In conclusion, Single Base C-sequencing of the first 500 bases of the 16S rRNA-gene could be a useful and widely applicable method for the identification of bacteria at the species level, with the added advantage of being more rapid and easier to automatize than full sequence determination.     

Comparison of different biochemical and molecular methods for the identification of Vibrio parahaemolyticus

AIMS: Multicentre evaluation of biochemical and molecular methods for the identification of Vibrio parahaemolyticus. METHODS AND RESULTS: For the biochemical identification methods, API 20E and API 20NE and Alsina's scheme were evaluated in intra- and interlaboratory tests in order to determine the accuracy and concordance of each method. Both in intra- and interlaboratory tests, the Alsina's scheme showed the highest sensitivity (86% of correct identifications in the interlaboratory test). False-positive results were obtained by all methods (specificity was 95% for API 20E, 73% for API 20NE and 84% for Alsina's scheme) and concordance varied from 65% of API 20NE to 84% of API 20E. For the molecular identifications, polymerase chain reaction (PCR) for the detection of toxR gene, tl gene and pR72H fragment were tested on 30 strains by two laboratories. The PCR for toxR showed the highest inclusivity (96%), exclusivity (100%) and concordance (97%). CONCLUSIONS: Among the biochemical identification methods tested, the Alsina's scheme gave more reliable results; however, in order to avoid false-positive results, all the biochemical identifications should be confirmed by means of molecular methods. SIGNIFICANCE AND IMPACT OF THE STUDY: Availability of an efficient identification method of Vibrio parahaemolyticus to use in official control of fisheries products.     

Nested ANOVA for Repeated Surveys with Autocorrelated Errors

Diet surveys in the science of human nutrition involve longitudinal observations on the same individuals repeated at distinct time points. Consequently a common feature of such surveys is inherent correlation between successive observations. In this paper, an autoregressive model of order 1 is assumed for the successive observations on the same individual in a survey. Further, observations on the same individual in two different surveys are assumed to be correlated. The technique of analysis of variance is modified suitably for this set up and applied to two nutritional surveys carried on army recruits.     

Validation of a PCR-based method for detection of food-borne thermotolerant campylobacters in a multicenter collaborative trial

A PCR-based method for rapid detection of food-borne thermotolerant campylobacters was evaluated through a collaborative trial with 12 laboratories testing spiked carcass rinse samples. The method showed an interlaboratory diagnostic sensitivity of 96.7% and a diagnostic specificity of 100% for chicken samples, while these values were 94.2 and 83.3%, respectively, for pig samples.     

Cluster analysis of potomac river survey stations based on protozoan presence-absence data

Four high-water and six low-water limnological surveys of a portion of the Potomac River were made from 1956 to 1965; samples were collected at three stations on each survey to determine the effects of operation of the PEPCO Dickerson Power Station on the aquatic biota. Cluster analyses were made of various combinations of Jaccard coefficients relating 46 aggregations of 647 protozoan species.Similarities of aggregations of species within a survey were nearly always greater than similarities among aggregations from different surveys, indicating linear or along-stream environmental influences. Within-survey similarities for the early and late surveys were usually higher than similarities within middle-year surveys, a possible indication of environmental change at all stations, including the control, and subsequent biotic readjustment. Clustering of the 1956 aggregations, taken under high-water conditions before plant operations began, with aggregations from other surveys for any one station indicate that 1956 may have been different from other survey years. One explanation is that increased urbanization upstream from the power station after 1956 caused some environmental change. No changes in aquatic biota could be attributed to thermal pollution as a direct result of operation of the electric power generating station.