Observer variation in field assessments of vegetation condition: Implications for biodiversity conservation

Summary   Uncertainty in assessments of vegetation condition that are used to inform land management and planning decisions for biodiversity conservation in Australia may lead to unexpected outcomes, including loss of biodiversity. This study investigates observer error in field estimates of vegetation attributes, one component of uncertainty in assessments of vegetation condition. Ten observers conducted vegetation condition assessments using two assessment protocols (BioMetric and Habitat Hectares) on 20 sites in a grassy woodland community. Observers' estimates varied substantially across multiple scoring categories for all vegetation attributes on almost all sites. Across all sites, the average coefficient of variation in total vegetation condition scores was 15–18% for both protocols, with a maximum of 60%. The primary cause of variation in total vegetation condition scores was random error in raw estimates of vegetation attributes, although sensitivity of some highly weighted attributes to error exacerbated variation in some cases. Observers generally agreed on the total scores and ranks of highly degraded (pasture) sites, but were less consistent on other sites. Rank correlations between pairs of observers were stronger for Habitat Hectares, suggesting BioMetric may be slightly more sensitive to observer error. It is recommended that: (i) research is undertaken into methods for reducing observer error; (ii) review is made of the sensitivity of index scoring structures to observer error; (iii) field observers estimate uncertainty around point estimates of vegetation condition; and, (iv) decision-makers explicitly incorporate uncertainty into the decision-making processes and aim for outcomes that are robust to this uncertainty.     

Evaluation of individual protein errors in silver-stained two-dimensional gels

The relationship between spot volume and variation for all protein spots observed on large format 2D gels when utilising silver stain technology and a model system based on mammalian NSO cell extracts is reported. By running multiple gels we have shown that the reproducibility of data generated in this way is dependent on individual protein spot volumes, which in turn are directly correlated with the coefficient of variation. The coefficients of variation across all observed protein spots were highest for low abundant proteins which are the primary contributors to process error, and lowest for more abundant proteins. Using the relationship between spot volume and coefficient of variation we show it is necessary to calculate variation for individual protein spot volumes. The inherent limitations of silver staining therefore mean that errors in individual protein spot volumes must be considered when assessing significant changes in protein spot volume and not global error.     

Negative staining and image classification — powerful tools in modern electron microscopy

Vitrification is the state-of-the-art specimen preparation technique for molecular electron microscopy (EM) and therefore negative staining may appear to be an outdated approach. In this paper we illustrate the specific advantages of negative staining, ensuring that this technique will remain an important tool for the study of biological macromolecules. Due to the higher image contrast, much smaller molecules can be visualized by negative staining. Also, while molecules prepared by vitrification usually adopt random orientations in the amorphous ice layer, negative staining tends to induce preferred orientations of the molecules on the carbon support film. Combining negative staining with image classification techniques makes it possible to work with very heterogeneous molecule populations, which are difficult or even impossible to analyze using vitrified specimens. Published: March 19, 2004.     

Interobserver variation in breast cancer grading: a statistical modeling approach

OBJECTIVE: To study random and the systematic error in breast cancer grading, to find the source of disagreements and measure the reliability of graders so that appropriate corrective action can be taken. STUDY DESIGN: Five independent observers graded 50 breast carcinoma slides from 50 consecutive breast cancer specimens according to the Nottingham criteria. The polychoric correlation was used to measure association. Stuart-Maxwell and McNemar tests were used to measure equality of thresholds. RESULTS: The polychoric correlation among observers was high (mean = 0.803, 0.712, 0.797 and 0.602 for the final grade, tubule formation, nuclear pleomorphism and mitotic figures, respectively). However, there were significant differences in thresholds (6, 7, 7 and 9 pairs of 10 showing significant differences in classification of grades/scores for final grade, tubule formation, nuclear pleomorphism and mitotic counts, respectively). CONCLUSION: The high polychoric correlations suggest that random error in grading breast cancers in this study was low, confirming the underlying reliability of grading and graders. However, significant differences in the thresholds lowers raw agreement. Such a scenario may be rectified by increased intradepartmental discussion.     

A model for the classification of specimens containing random proportions of abnormal cells

The effect of abnormal cell proportion on the performance of an automated cervical prescreening system is discussed in K.R. Castleman and B.S. White, Cytometry 2:155-18. The model employed assumes fixed proportions of abnormal cells, both in the design stage and in the test stage. In the present paper, an extended model is developed that allows for random variability of this proportion. It is shown, that there is a fundamental, non-zero lower limit to the false-negative specimen error rate, which depends only on the coefficient of variation. This limit may be reached even for moderate values of the coefficient of variation, which implies that a satisfactory prescreening system may not be feasible.     

Hybrid bounded error observers for uncertain bioreactor models

In this paper, we build bounded error observers for a common class of partially known bioreactor models. The main idea is to construct hybrid bounded observers “between” high gain observer, which has an adjustable convergence rate but requires perfect knowledge of the model, and asymptotic observer which is very robust towards uncertainty but has a fixed convergence rate. An hybrid bounded error observer which reconstructs the two state variables is constructed considering two steps: first step is similar to a high gain observer meaning that fast convergence rate but error depending on the knowledge of the model are obtained; second step is a switch to an observer similar to the asymptotic one meaning that fixed convergence rate towards an error as small as desired is obtained. Thus, a better convergence rate of estimated variables than the classical asymptotic observer is obtained.     

Identification of insulin receptor systems: assessing the impact of model selection and measurement error on precision of parameter estimates using Monte Carlo study

An extensive Monte Carlo study has been carried out in order to study the effect of measurement error on the precision of parameter estimates of an insulin binding system. Hypothetical radioimmunoassay experiments were generated for insulin binding to erythrocytes. The design of experiments followed strictly the protocol of real experiments. Randomly generated error was added to the synthetic data. The standard technique, a weighted non-linear regression analysis, was employed to re-estimate parameters of a model of two receptor sites and a model of negative co-operativity. As the original parameter values were known, the differences between original and estimated values was studied for (a) measurement error in the range from 0-17%, (b) random initial estimates and (c) error-free non-specific binding. In addition, analytical estimates of parameter precision were compared with the true between-experiment variation of parameter estimates. At the measurement error of 12%, a one site model is recommended to estimate the high affinity population of the two sites model. Plausible results can be expected in 90% of experiments, the between-experiment variation being approximately 30%. The model of two receptor sites gives approximately two thirds of plausible results. The high affinity population can be estimated with the between-experiment variation of 40%, the low affinity population is virtually unidentifiable with the between-experiment variation of approximately 100% and parameter estimates biased to higher values. Only half of the results obtained from the model of negative co-operativity are plausible, the variation in parameter estimates ranges from 90-150% and estimates are biased to higher values. At the level of 12% measurement error, random initial estimates do not significantly affect the estimation process, provided initial estimates are selected from a feasible range. At the same measurement error, the error-free non-specific binding does not improve the results, indicating that the mean of six replicates may be taken as a reliable estimate of non-specific binding. The analytical estimates of the coefficient of variation systematically underestimates the true between-experiments coefficient of variation, the difference has been found to be about 50%.     

A comparison of species diversity estimators

Although having been much criticized, diversity indices are still widely used in animal and plant ecology to evaluate, survey, and conserve ecosystems. It is possible to quantify biodiversity by using estimators for which statistical characteristics and performance are, as yet, poorly defined. In the present study, four of the most frequently used diversity indices were compared: the Shannon index, the Simpson index, the Camargo eveness index, and the Pielou regularity index. Comparisons were performed by simulating the Zipf–Mandelbrot parametric model and estimating three statistics of these indices, i.e., the relative bias, the coefficient of variation, and the relative root-mean-squared error. Analysis of variance was used to determine which of the factors contributed most to the observed variation in the four diversity estimators: abundance distribution model or sample size. The results have revealed that the Camargo eveness index tends to demonstrate a high bias and a large relative root-mean-squared error whereas the Simpson index is least biased and the Shannon index shows a smaller relative root-mean-squared error, regardless of the abundance distribution model used and even when sample size is small. Shannon and Pielou estimators are sensitive to changes in species abundance pattern and present a nonnegligible bias for small sample sizes (<1000 individuals). Received: May 8, 1998 / Accepted: May 6, 1999     

Observer error in vegetation surveys: a review

Aims Vegetation sampling employing observers is prone to both inter-observer and intra-observer error. Three types of errors are common: (i) overlooking error (i.e. not observing species actually present), (ii) misidentification error (i.e. not correctly identifying species) and (iii) estimation error (i.e. not accurately estimating abundance). I conducted a literature review of 59 articles that provided quantitative estimates or statistical inferences regarding observer error in vegetation studies.Important findings Almost all studies (92%) that tested for a statistically significant effect of observer error found at least one significant comparison. In surveys of species composition, mean pseudoturnover (the percentage of species overlooked by one observer but not another) was 10–30%. Species misidentification rates were on the order of 5–10%. The mean coefficient of variation (CV) among observers in surveys of vegetation cover was often several hundred % for species with low cover, although CVs of 25–50% were more representative of species with mean covers of>50%. A variety of metrics and indices (including commonly used diversity indices) and multivariate data analysis techniques (including ordinations and classifications) were found to be sensitive to observer error. Sources of error commonly include both characteristics of the vegetation (e.g. small size of populations, rarity, morphology, phenology) and attributes of the observers (e.g. mental fatigue, personal biases, differences in experience, physical stress). The use of multiple observers, additional training including active feedback approaches, and continual evaluation and calibration among observers are recommended as strategies to reduce observer error in vegetation surveys.     

Interobserver and intraobserver variability in the cytologic diagnosis of normal and abnormal metaplastic squamous cells in pap smears

OBJECTIVE: Interoberver variability has important implications for patient care, diagnostic error and medical litigation. In the management of any cervical epithelial abnormality, its biologic significance as well as diagnostic reproducibility is very important. Interobserver variability has not been measured adequately for metaplastic squamous lesions. We analyzed interobserver and intraobserver variability and diagnostic accuracy in the diagnosis of dysplastic metaplastic cells. STUDY DESIGN: Sixty Pap smears from patients with abnormalities of metaplastic squamous cells of varying severity were selected from the files of Lankenau Hospital, Wynnewood, Pennsylvania, U.S.A., diagnosed between 1990 and 1996. These were reviewed by four observers with different levels of cytology experience. Each of the observers blindly and independently reviewed all Pap smears. Tabulated results were analyzed to determine interobserver and intraobserver variability and diagnostic accuracy. RESULTS: Statistically significant interobserver reproducibility was found between both inexperienced observers as well as between observers 1 (experienced) and 3 (inexperienced) and between observers 2 (experienced) and 4 (inexperienced). The observed degree of agreement between both experienced observers (1 and 2) reflected random rating rather than reproducibility. There was no difference in interobserver reproducibility in low vs. high grade lesions. Intraobserver reproducibility had no significant correlation with experience of the observer. The sensitivity ranged from 0.69 to 0.97 (mean, 0.79), while the specificity ranged from 0.09 to 0.46 (mean, 0.30). Mean diagnostic accuracy was better in benign and low grade squamous intraepithelial lesions in comparison to high grade squamous intraepithelial lesions. CONCLUSION: There was good interobserver agreement in classifying squamous metaplastic lesions. The agreement did not correlate with grade of dysplasia or experience of the cytopathologists. These findings should be considered in making treatment, quality assurance and legal decisions. A larger study is indicated to study interobserver and intraobserver variability and define cytologic criteria for lesions of metaplastic squamous cells.     

Exploration of methods to identify polymorphisms associated with variation in DNA repair capacity phenotypes

Elucidating the relationship between polymorphic sequences and risk of common disease is a challenge. For example, although it is clear that variation in DNA repair genes is associated with familial cancer, aging and neurological disease, progress toward identifying polymorphisms associated with elevated risk of sporadic disease has been slow. This is partly due to the complexity of the genetic variation, the existence of large numbers of mostly low frequency variants and the contribution of many genes to variation in susceptibility. There has been limited development of methods to find associations between genotypes having many polymorphisms and pathway function or health outcome. We have explored several statistical methods for identifying polymorphisms associated with variation in DNA repair phenotypes. The model system used was 80 cell lines that had been resequenced to identify variation; 191 single nucleotide substitution polymorphisms (SNPs) are included, of which 172 are in 31 base excision repair pathway genes, 19 in 5 anti-oxidation genes, and DNA repair phenotypes based on single strand breaks measured by the alkaline Comet assay. Univariate analyses were of limited value in identifying SNPs associated with phenotype variation. Of the multivariable model selection methods tested: the easiest that provided reduced error of prediction of phenotype was simple counting of the variant alleles predicted to encode proteins with reduced activity, which led to a genotype including 52 SNPs; the best and most parsimonious model was achieved using a two-step analysis without regard to potential functional relevance: first SNPs were ranked by importance determined by random forests regression (RFR), followed by cross-validation in a second round of RFR modeling that included ever more SNPs in declining order of importance. With this approach six SNPs were found to minimize prediction error. The results should encourage research into utilization of multivariate analytical methods for epidemiological studies of the association of genetic variation in complex genotypes with risk of common diseases.     

Pollen stainability variation in the apomictic blackberry, Rubus nessensis (Rosaceae)

Pollen quality was assessed with cotton blue staining in an apomictic blackberry, Rubus nessensis W. Hall, on plants grown outdoors in a random arrangement under uniform conditions. Significant variation between populations may be attributed mainly to genetical factors. Most of the variation found between plants, sometimes even between ramets, within a population must, however, be environmentally caused, likewise the variation between flowers within the same plant.     

Size distribution of mouse Langerhans islets

Pancreatic beta-cells are clustered in islets of Langerhans, which are typically a few hundred micrometers in a variety of mammals. In this study, we propose a theoretical model for the growth of pancreatic islets and derive the islet size distribution, based on two recent observations: First, the neogenesis of new islets becomes negligible after some developmental stage. Second, islets grow via a random process, where any cell in an islet proliferates with the same rate regardless of the present size of the islet. Our model predicts either log-normal or Weibull distributions of the islet sizes, depending on whether cells in an islet proliferate coherently or independently. To confirm this, we also measure the islet size by selectively staining islets, which are exposed from exocrine tissues in mice after enzymatic treatment. Indeed revealed are skewed distributions with the peak size of approximately 100 cells, which fit well to the theoretically derived ones. Interestingly, most islets turned out to be bigger than the expected minimal size (approximately 10 or so cells) necessary for stable synchronization of beta-cells through electrical gap-junction coupling. The collaborative behavior among cells is known to facilitate synchronized insulin secretion and tends to saturate beyond the critical (saturation) size of approximately 100 cells. We further probe how the islets change as normal mice grow from young (6 weeks) to adult (5 months) stages. It is found that islets may not grow too large to maintain appropriate ratios between cells of different types. Our results implicate that growing of mouse islets may be regulated by several physical constraints such as the minimal size required for stable cell-to-cell coupling and the upper limit to keep the ratios between cell types. Within the lower and upper limits the observed size distributions of islets can be faithfully regenerated by assuming random and uncoordinated proliferation of each beta-cell at appropriate rates.     

The interaction of genetic drift and mutation with selection in a fluctuating environment

The interaction of genetic drift, mutation, and selection in a random environment is investigated using an asymptotic analysis based on assumptions of weak mutation and strong selection. It is shown that genetic drift can be a potent force for removing variation from the population when the random environment tends to occasionally push alleles down to low frequencies.     

PROBLEMS IN RADIOLOGICAL INTERPRETATION

Recent comparative x-ray studies demonstrate: (1) Photofluorography is a relatively efficient tool for tuberculosis case finding. (2) All photofluorograms should be interpreted independently by two competent interpreters. (3) The activity of a lesion cannot be determined from a single roentgenogram.There is a variation in film interpretation between different observers, similar to that noted in many other fields which have been subjected to valid test. This variation is now the source of extensive investigations, out of which should come considerable progress in all branches of medicine.     

An application of a mixed-effects location scale model for analysis of Ecological Momentary Assessment (EMA) data

Summary .   For longitudinal data, mixed models include random subject effects to indicate how subjects influence their responses over repeated assessments. The error variance and the variance of the random effects are usually considered to be homogeneous. These variance terms characterize the within-subjects (i.e., error variance) and between-subjects (i.e., random-effects variance) variation in the data. In studies using ecological momentary assessment (EMA), up to 30 or 40 observations are often obtained for each subject, and interest frequently centers around changes in the variances, both within and between subjects. In this article, we focus on an adolescent smoking study using EMA where interest is on characterizing changes in mood variation. We describe how covariates can influence the mood variances, and also extend the standard mixed model by adding a subject-level random effect to the within-subject variance specification. This permits subjects to have influence on the mean, or location, and variability, or (square of the) scale, of their mood responses. Additionally, we allow the location and scale random effects to be correlated. These mixed-effects location scale models have useful applications in many research areas where interest centers on the joint modeling of the mean and variance structure.     

In the eye of the beholder: bias and stochastic variation in cover estimates

Cover estimates by eye is a prevailing method to assess abundance. We examined cover estimates with regard to bias and random variation. Ten observers working with a national forest vegetation survey estimated sixteen 100 m²-plots, placed in two different vegetation types. These had similar species composition but were clearly distinguishable in the field. In species-wise analyses, observer bias varied greatly, with Dicranum spp., Vaccinium vitis-idaea and Vaccinium myrtillus having the largest bias. Experience had a surprisingly small impact on variation. Power analysis revealed only small differences between observers in the ability to distinguish the two vegetation types, and little value in averaging the assessments from two, three or four observers. Cover estimates did better than presence/absence data in separating the two vegetation types and multivariate analyses were more powerful than univariate ones.     

Analogous mechanisms compensate for neural delays in the sensory and the motor pathways: evidence from motor flash-lag

Motor behaviors require animals to coordinate neural activity across different areas within their motor system. In particular, the significant processing delays within the motor system must somehow be compensated for. Internal models of the motor system, in particular the forward model, have emerged as important potential mechanisms for compensation. For motor responses directed at moving visual objects, there is, additionally, a problem of delays within the sensory pathways carrying crucial position information. The visual phenomenon known as the flash-lag effect has led to a motion-extrapolation model for compensation of sensory delays. In the flash-lag effect, observers see a flashed item colocalized with a moving item as lagging behind the moving item. Here, we explore the possibility that the internal forward model and the motion-extrapolation model are analogous mechanisms compensating for neural delays in the motor and the visual system, respectively. In total darkness, observers moved their right hand gripping a rod while a visual flash was presented at various positions in relation to the rod. When the flash was aligned with the rod, observers perceived it in a position lagging behind the instantaneous felt position of the invisible rod. These results suggest that compensation of neural delays for time-varying motor behavior parallels compensation of delays for time-varying visual stimulation.     

Comparative methods based on species mean values

Comparative methods that use simple linear regression based on species mean values introduce three difficulties with respect to the standard regression model. First, species values may not be independent because they form part of a hierarchically structured phylogeny. Second, variation about the regression line includes two sources of error: 'biological error' due to deviations of the true species mean values from the regression line and sampling error associated with the estimation of these mean values [B. Riska, Am. Natural. 138 (1991) 283]. Third, sampling error in the independent variable results in an attenuated estimate of the regression slope. We consider estimation and hypothesis testing using two statistical models which explicitly justify the use of the species mean values, without the need to account for phylogenetic relationships. The first (random-effects) is based on an evolutionary model whereby species evolve to fill a bivariate normal niche space, and the second (fixed-effects) is concerned with describing a relationship among the particular species included in a study, where the only source of error is in the estimation of species mean values. We use a modification of the maximum-likelihood method to obtain an unbiased estimate of the regression slope. For three real datasets we find a close correspondence between this slope and that obtained by simply regressing the species mean values on each other. In the random effects model, the P-value also approximates that based on the regression of species mean values. In the fixed effects model, the P-value is typically much lower. Simulated examples illustrate that the maximum-likelihood approach is useful when the accuracy in estimating the species mean values is low, but the traditional method based on a regression of the species mean values may often be justified provided that the evolutionary model can be justified.