Bad Reporting or Bad Science? Systematic Data Evaluation as a Means to Improve the Use of Peer-Reviewed Studies in Risk Assessments of Chemicals

In this study we assess the applicability of a set of reliability criteria proposed by Ågerstrand et al. This was done by evaluating the reliability of 12 non-standard peer-reviewed ecotoxicity and toxicity studies for Bisphenol A. There was an overall agreement between the evaluator and the authors of the papers regarding the result of the evaluations. This suggests that the criteria offer enough guidance to be a useful and consistent evaluation tool. It provides a transparent and structured approach, and ensures that a minimum and similar set of criteria is used. The evaluation of the peer-reviewed ecotoxicity and toxicity studies concludes that important information is sometimes missing, and therefore the studies do not always meet common regulatory requirements regarding reporting. Whether this is due to insufficient reporting or due to poorly performed studies is not known. To improve the reporting, and thereby promote reliability and reproducibility, researchers, reviewers, and editors are recommended to use the suggested criteria as a guideline. In conclusion, in order to improve the reliability of peer-reviewed studies, and to increase their use in regulatory risk assessments of chemicals, the dialog between regulators, researchers, and editors regarding how to evaluate and report studies needs to be strengthened.     

Qualichem In Vivo: A Tool for Assessing the Quality of In Vivo Studies and Its Application for Bisphenol A

In regulatory toxicology, quality assessment of in vivo studies is a critical step for assessing chemical risks. It is crucial for preserving public health studies that are considered suitable for regulating chemicals are robust. Current procedures for conducting quality assessments in safety agencies are not structured, clear or consistent. This leaves room for criticism about lack of transparency, subjective influence and the potential for insufficient protection provided by resulting safety standards. We propose a tool called “Qualichem in vivo” that is designed to systematically and transparently assess the quality of in vivo studies used in chemical health risk assessment. We demonstrate its use here with 12 experts, using two controversial studies on Bisphenol A (BPA) that played an important role in BPA regulation in Europe. The results obtained with Qualichem contradict the quality assessments conducted by expert committees in safety agencies for both of these studies. Furthermore, they show that reliance on standardized guidelines to ensure scientific quality is only partially justified. Qualichem allows experts with different disciplinary backgrounds and professional experiences to express their individual and sometimes divergent views—an improvement over the current way of dealing with minority opinions. It provides a transparent framework for expressing an aggregated, multi-expert level of confidence in a study, and allows a simple graphical representation of how well the study integrates the best available scientific knowledge. Qualichem can be used to compare assessments of the same study by different health agencies, increasing transparency and trust in the work of expert committees. In addition, it may be used in systematic evaluation of in vivo studies submitted by industry in the dossiers that are required for compliance with the REACH Regulation. Qualichem provides a balanced, common framework for assessing the quality of studies that may or may not be following standardized guidelines.     

Verification of a REACH Environmental Prioritization System Against Regulatory Risk Indices

A prioritization method that was developed to rank chemical substances on the basis of their environmental impact was applied to 230 new substance notifications from earlier European chemical legislation (67/548/EEC). The method encompasses three steps: (1) assigning an environmental hazard score, (2) assigning an environmental exposure score, and (3) combining these two scores into one priority score. In this study, the resulting scores ranged between 4 (highest priority) and 15 (lowest priority). The scores were compared with results from risk assessments available for 138 of the 230 substances. For most substances in these assessments, a priority score of 12 or higher was associated with no or limited risk, while a score of 11 or lower represented high risk or led to the conclusion that risk reduction measures were required. This categorization applied to all but 15 of the 138 substances. The method was also used for ranking the first 15 Substances of Very High Concern (SVHC) under REACH regulation, to compare the priority scores of new substances to those of the SVHC. In sum, the prioritization method seems to be valuable to identify substances of concern with respect to the environment.     

In vitro tests within the REACH information strategies

Tonnage-based information requirements are specified in the proposal on the regulation on the Registration, Evaluation and Authorisation of Chemicals (REACH) in the European Union. The hazard assessment for toxic endpoints should be performed by using a tiered approach, i.e. as an information strategy (IS), starting with an evaluation of all of the data already available, including animal in vivo and in vitro data, and human evidence and case reports, as well as data from (Quantitative)-Structure Activity Relationships ([Q]SARs) or read-across, before any further testing is suggested. To contribute to the implementation of the REACH system, the Nordic countries launched two projects: 1) a review of currently used testing strategies, including a comparison with the REACH requirements; and 2) the development of detailed ISs for skin and eye irritation/corrosion. The review showed that the ISs and classification criteria for the selected endpoints are inconsistent in many cases. In the classification criteria, human data and in vivo test results are usually the prerequisites. Other types of information, such as data from in vitro studies, can sometimes be used, but usually as supportive evidence only. This differs from the REACH ISs, where QSARs, read-across and in vitro testing are important elements. In the other part of the project, an IS for skin and eye irritation/corrosion was proposed. The strategy was "tested" by using four high production volume (HPV) chemicals: hydrogen peroxide, methyl tertiary-butyl ether (MTBE), trivalent chromium, and diantimony trioxide, but only MTBE and trivalent chromium are dealt with in this paper. The "test" revealed that in vivo data, human case reports and physical-chemical data were available and could be used in the evaluation. Classification could be based on the proposed IS and the existing data in all cases, except for the eye irritation/corrosion of trivalent chromium. Weight-of-evidence analysis appeared to be a useful step in the ISs proposed, and including it in the REACH strategies should be considered. For these chemicals, few in vitro and (Q)SAR data were available--more of these data would be generated, if the relevant guidance and legislation on classification were updated.     

Redlistr: tools for the IUCN Red Lists of ecosystems and threatened species in R

The International Union for the Conservation of Nature (IUCN) Red List of ecosystems and Red List of threatened species are global standards for assessing risks of ecosystem collapse and species extinction. However, misconceptions of the Red List assessment process, along with its technically demanding nature, can result in the misapplication of their criteria, leading to inconsistent and potentially unreliable assessments. To address this problem, we developed redlistr, an R package aiding in the production of consistent species and ecosystem Red List assessments. Redlistr's features include methods to calculate 1) area from spatial data, 2) range size metrics, 3) rates of change of distributions or populations, and 4) distribution or population at another time from these rates. A key feature of the package is the systematic approach used to eliminate geometric uncertainty when estimating area of occupancy. Here, we develop two case studies to demonstrate the functionalities of redlistr with typical workflows for both species and ecosystems. Redlistr was developed to be accessible to users with a broad range of experience in programming for spatial and temporal data analysis, and sufficiently flexible to allow users to parameterise functions and select equations to fit their purposes. The package specifically aims to assist researchers and conservation practitioners to conduct robust and transparent risk assessments of ecosystems and species under the IUCN Red List criteria but is also useful for other studies requiring analyses of range size, area change and calculations of rates of change.     

Evaluating Chemical Interaction Studies for Mixture Risk Assessment

We describe a set of criteria to evaluate the quality of data and interpretations in chemical interaction studies. These criteria reflect the consensus of the literature on interaction analysis developed over decades of research in pharmacology, toxicology, and biometry; address common pitfalls in published interaction studies; and can be easily applied to common methods of interaction analysis. The criteria apply broadly to interaction data for drugs, pesticides, industrial chemicals, food additives, and natural products and are intended to assist risk assessors who must evaluate interaction studies for use in component-based mixture risk assessments. The criteria may also assist researchers interested in conducting interaction studies to inform mixture risk assessment. The criteria are also intended to serve larger scientific goals, including increasing the repeatability of results obtained in chemical interaction studies, enhancing the reliability of conclusions drawn from interaction data, providing greater consistency of interpretations among various analysts, and decreasing uncertainty in using interaction data in risk assessments. We describe the basis for each criterion and demonstrate their utility by using them to evaluate interaction studies from the recent toxicological and pharmacological literature, which serve as examples of different types of data sets that the risk assessor may encounter.     

数学形态学在昆虫分类学上的应用研究.Ⅰ.在目级阶元上的应用研究

根据昆虫图像,对半翅目（Hemiptera）、鳞翅目（Lepidoptera）、鞘翅目（Coleoptera）的34种昆虫提取形状参数、叶状性、球状性等7项数学形态特征进行了统计分析,从而论证了各项数学形态特征在目级昆虫分类阶元上作为分类特征的可行性和可靠性,并从数学形态学角度对所涉及到的同阶元昆虫类群的亲缘关系做了描述。结果表明,在作为目级阶元分类特征时,各项特征的可靠性依次为：(似圆度、偏心率、亮斑数)＞(叶状性、球状性、圆形性)＞形状参数。由这些特征的差异显著性可知,从数学形态特征角度讲,3个目的亲缘关系远近大小依次为：半翅目与鞘翅目＞半翅目与鳞翅目＞鳞翅目与鞘翅目。     

New tools for systematic evaluation of teaching qualities of medical faculty: results of an ongoing multi-center survey

Background

Tools for the evaluation, improvement and promotion of the teaching excellence of faculty remain elusive in residency settings. This study investigates (i) the reliability and validity of the data yielded by using two new instruments for evaluating the teaching qualities of medical faculty, (ii) the instruments'' potential for differentiating between faculty, and (iii) the number of residents'' evaluations needed per faculty to reliably use the instruments.

Methods and Materials

Multicenter cross-sectional survey among 546 residents and 629 medical faculty representing 29 medical (non-surgical) specialty training programs in the Netherlands. Two instruments—one completed by residents and one by faculty—for measuring teaching qualities of faculty were developed. Statistical analyses included factor analysis, reliability and validity exploration using standard psychometric methods, calculation of the numbers of residents'' evaluations needed per faculty to achieve reliable assessments and variance components and threshold analyses.

Results

A total of 403 (73.8%) residents completed 3575 evaluations of 570 medical faculty while 494 (78.5%) faculty self-evaluated. In both instruments five composite-scales of faculty teaching qualities were detected with high internal consistency and reliability: learning climate (Cronbach''s alpha of 0.85 for residents'' instrument, 0.71 for self-evaluation instrument, professional attitude and behavior (0.84/0.75), communication of goals (0.90/0.84), evaluation of residents (0.91/0.81), and feedback (0.91/0.85). Faculty tended to evaluate themselves higher than did the residents. Up to a third of the total variance in various teaching qualities can be attributed to between-faculty differences. Some seven residents'' evaluations per faculty are needed for assessments to attain a reliability level of 0.90.

Conclusions

The instruments for evaluating teaching qualities of medical faculty appear to yield reliable and valid data. They are feasible for use in medical residencies, can detect between-faculty differences and supply potentially useful information for improving graduate medical education.     

New method and criteria for national assessments of threatened habitat types

Consistent information on threatened habitat types is needed for land use planning and for prioritizing conservation, management, and restoration actions. However, detailed background data for assessing extinction risks of habitat types exists only in few countries. We present a new, flexible procedure for assigning habitat types into Red List Categories similar to those used for species by the World Conservation Union (IUCN). The procedure allows variation in the character or scale of assessment units and it is applicable even with incomplete data. The assessment protocol consists of two primary criteria: the change in the quantity and the change in the quality of the habitat type. The criteria are analyzed by expert groups with a transparent and repeatable stepwise procedure. The quantitative and qualitative changes in habitat types over the last 50 years serve as a starting point for the assessment, and the status is adjusted by assessing sub-criteria that address earlier changes, predicted future change, and the overall commonness or rarity of the habitat type. We also report the main results of the first assessment of threatened habitat types in Finland, and illustrate the application of the criteria by two case studies.     

A structured analysis of risk to important wildlife elements in three Australian Wildlife Conservancy sanctuaries

Where data and information are lacking, structured expert risk assessments can provide a powerful tool to progress natural resource planning. In many situations, practitioners make informal assessments of risk within small groups that typically constitute employees. In this study, we report on three small (in terms of experts) structured expert‐based risk assessment case studies conducted by expert employees of a not‐for‐profit organisation (Australian Wildlife Conservancy) to demonstrate the utility of the approach. The case studies were carried out for three wildlife sanctuaries managed by AWC: Faure Island, Karakamia and Paruna. The likelihood that a set of direct risk factors would cause management failure for sets of important wildlife elements in the three sanctuaries was elicited from the small group of ecological experts. The analysis was couched in terms of a management aim to not lose species from each wildlife element over the management period of 25 years with current management. The experts believed, in particular, that increasing temperature and decreasing water availability associated with climate change were likely to impact significantly upon the vegetation elements and water‐reliant fauna associated with the sanctuaries. Some vegetation elements were also thought likely to be at risk of over‐grazing, unsuitable fire regimes and, in some cases, disease. In addition to predation by exotic predators at one sanctuary, the experts identified additional direct risk factors for various fauna elements associated with expected changes to the vegetation elements, including reduction in food availability, nesting habitat, and generally important life media. From the risk analyses, a preliminary conceptual model was developed to underpin monitoring and to indicate areas for possible management intervention and research. The case studies demonstrate that even in a small workplace team, structured risk assessments can be efficiently accomplished and can provide expedient and transparent information that effectively captures and aggregates the views of the experts.     

Comparing functional annotation analyses with Catmap

Background  

Ranked gene lists from microarray experiments are usually analysed by assigning significance to predefined gene categories, e.g., based on functional annotations. Tools performing such analyses are often restricted to a category score based on a cutoff in the ranked list and a significance calculation based on random gene permutations as null hypothesis.     

Reliability of a computational platform as a surrogate for manually interpreted immunohistochemical markers in breast tumor tissue microarrays

BackgroundPathologist and computational assessments have been used to evaluate immunohistochemistry (IHC) in epidemiologic studies. We compared Definiens Tissue Studio® to pathologist scores for 17 markers measured in breast tumor tissue microarrays (TMAs) [AR, CD20, CD4, CD8, CD163, EPRS, ER, FASN, H3K27, IGF1R, IR, Ki67, phospho-mTOR, PR, PTEN, RXR, and VDR].Methods5 914 Nurses’ Health Study participants, diagnosed 1976–2006 (NHS) and 1989–2006 (NHS-II), were included. IHC was conducted by the Dana-Farber/Harvard Cancer Center Specialized Histopathology Laboratory. The percent of cells staining positive was assessed by breast pathologists. Definiens output was used to calculate a weighted average of percent of cells staining positive across TMA cores for each marker. Correlations between pathologist and computational scores were evaluated with Spearman correlation coefficients. Receiver-operator characteristic curves were constructed, using pathologist scores as comparison.ResultsSpearman correlations between pathologist and Definiens assessments ranged from weak (RXR, rho=-0.05; CD163, rho = 0.10) to strong (Ki67, rho = 0.79; pmTOR, rho = 0.77). The area under the curve was >0.70 for all markers except RXR.ConclusionOur data indicate that computational assessments exhibit variable correlations with interpretations made by an expert pathologist, depending on the marker evaluated. This study provides evidence supporting the use of computational platforms for IHC evaluation in large-scale epidemiologic studies, with the caveat that pilot studies are necessary to investigate agreement with expert assessments. In sum, computational platforms may provide greater efficiency and facilitate high-throughput epidemiologic analyses.     

数学形态学在昆虫分类学上的应用研究.Ⅰ.在目级阶元上的应用研究

根据昆虫图像,对半翅目（Hemiptera）、鳞翅目（Lepidoptera）、鞘翅目（Coleoptera）的34种昆虫提取形状参数、叶状性、球状性等7项数学形态特征进行了统计分析,从而论证了各项数学形态特征在目级昆虫分类阶元上作为分类特征的可行性和可靠性,并从数学形态学角度对所涉及到的同阶元昆虫类群的亲缘关系做了描述。结果表明,在作为目级阶元分类特征时,各项特征的可靠性依次为：(似圆度、偏心率、亮斑数)＞(叶状性、球状性、圆形性)＞形状参数。由这些特征的差异显著性可知,从数学形态特征角度讲,3个目的亲缘关系远近大小依次为：半翅目与鞘翅目＞半翅目与鳞翅目＞鳞翅目与鞘翅目。     

Defining a role for herbarium data in Red List assessments: a case study of Plectranthus from eastern and southern tropical Africa

Red Lists are widely used to indicate species at risk of extinction. Specimen sheets in herbaria provide an important source of data relevant for Red List assessments. The aims of this paper are to establish which data can be sourced from specimen information to satisfy IUCN Red Data List criteria and to identify the specific criteria that can be used. Red List parameters are measured within a Geographical Information System (GIS), as this provides an objective and repeatable methodology which is less subjective than manual methods. Data used to explore this were gathered during the course of preparing a monograph on Plectranthus (Lamiaceae). Criteria relating to distribution (extent of occurrence, area of occupancy and fragmentation) and population profile (projected continuing decline and number of subpopulations) proved most suitable for assigning categories of threat. Estimates of mature individuals, generation length, population size, population reduction, extreme fluctuation and number of locations could not be derived from herbarium material without making inconsistent subjective decisions. In addition to comprehensively databased specimen information, extensive field knowledge is required to produce better estimates for assessing extinction risk. In order to enhance the usefulness of specimen information in the future, improvements in recording additional botanical data at the time of collection would be beneficial. Overall, herbaria provide a useful starting point for conservation-related work and can help to guide future work.     

Reliability of a structured method of selecting abstracts for a plastic surgical scientific meeting

There is no generally accepted method for assessing abstracts that are submitted for a medical scientific meeting. This article describes the development and prospective evaluation of such a method applied to the 220 abstracts submitted for the 2000 Annual Meeting of the European Association of Plastic Surgeons. Structured abstracts were evaluated in three categories: aesthetic surgery, basic research, and clinical study. Each anonymous abstract was assessed separately by 10 reputable European plastic surgeons. These reviewers used a structured rating questionnaire which resulted in a score given by each reviewer to each abstract between -6 and +6. The scores of all 10 reviewers were added for each abstract, and the papers were accepted in each of the three categories on the basis of this abridged score. To evaluate the reliability of this structured method of selection, the interrater agreement among the reviewers was tested by means of kappa analysis and the Cronbach alpha coefficient. The kappa values for agreement among reviewers regarding acceptability of abstracts were low, but the alpha coefficient indicated an acceptable degree of reliability of the average reviewers' ratings for all categories. Using a structured questionnaire can be helpful in the objective assessment of abstracts for a scientific meeting and may facilitate comparison of abstracts. Meritocratic dichotomy of abstracts by the reviewers is advocated to further improve reliability of the rating. Even though reliability generally increases with the number of reviewers, the annual increase of submitted abstracts may necessitate a decrease in the number of reviewers for each abstract.     

Inter-rater agreement of assessed prenatal maternal occupational exposures to lead

BACKGROUND: Industrial hygienists' assessments of prenatal occupational exposures based on parental job histories is a promising approach for population-based case-control studies of birth defects and other perinatal outcomes. However, evaluations of inter-rater agreement of such assessments have been limited. METHODS: We examined inter-rater agreement of occupational lead exposure assessments of maternal job reports by industrial hygienists in a population-based case-control study of parental occupational lead exposure and low birth weight. A total of 178 jobs with potential exposure to lead during the 6 months before pregnancy to the end of pregnancy were examined. Three industrial hygienists evaluated these jobs independently for exposure to lead including probability of exposure, type of exposure, route of entry, exposure frequency, duration, and intensity. Inter-rater agreement of these assessments beyond chance was evaluated using the kappa statistic (kappa). RESULTS: In general, inter-rater agreement was greater for assessment of direct exposures than assessment of indirect exposures. However, inter-rater agreement varied with the lead exposure metric under consideration, being: 1) fair to good for type of direct exposure (i.e., inorganic or organic), respiratory exposure and frequency of exposure to direct inorganic lead, hours per day of direct (i.e., inorganic or organic), and intensity of direct inorganic exposure; 2) poor for probability and type of indirect exposure (inorganic or organic); and 3) indeterminate for frequency of direct organic exposure, frequency of indirect exposures (organic or inorganic), and intensity of direct exposures (organic or inorganic). CONCLUSION: Retrospective assessment of maternal prenatal exposures to lead by industrial hygienists can provide some reliable metrics of exposure for studies of perinatal outcomes. Reliability studies of such exposure assessments may be useful for: quantifying the reliability of derived exposure metrics; identifying exposure metrics for exposure-outcome analyses; and determining the reliability of prenatal occupational exposures to other agents of interest.     

Minimizing Environmental Risks Using Fuzzy TOPSIS: Location Selection for the ITU Faculty of Management

ABSTRACT

Environmental risk is a measure of potential threats to the environment and combines the probability that events will cause or lead to degradation of the environment and the severity of that degradation. Evaluation of environmental risk factors requires linguistic terms to be used rather than exact numerical assignments. The fuzzy set theory provides a useful tool for converting linguistic terms into numerical evaluations. In this article, we will try to select the best location for the Faculty of Management of the Istanbul Technical University taking care of the predetermined environmental criteria by using fuzzy TOPSIS. The decision matrix is composed of linguistic evaluations for the alternatives with respect to the environmental risk criteria. The Macka Campus area has been selected after the multicriteria evaluation. A sensitivity analysis is also realized to see if the minor changes in criteria weights cause any change in the ranking of alternatives. Unless the importance of the criteria rock or soil structure, remoteness to health facilities, transportation availability, and transportation costs is changed to be from low to extremely low importance, the Macka Campus is still the selected alternative.