What are we measuring? Novices agree amongst themselves (but not always with experts) in their assessment of dog behaviour

Humans differ in how they perceive, assess, and measure animal behaviour. This is problematic because strong observer bias can reduce statistical power, accuracy of scientific inference, and in the worst cases, lead to spurious results. Unfortunately, reports and studies of measurement reliability in animal behaviour studies are rare. Here, we investigated two aspects of measurement reliability in working dogs: inter‐observer agreement and criterion validity (comparing novice ratings with those given by experts). Here, we extend for the first time a powerful framework used in human psychological studies to investigate three potential aspects of (dis)agreement in nonhuman animal behaviour research: (a) that some behaviours are easier to observe than others; (b) that some subjects are easier to observe than others; and (c) that observers with different levels of experience with the subject animal give the same or different ratings. We found that novice observers with the same level of experience agreed upon measures of a wide range of behaviours. We found no evidence that age of the dogs affected agreement between these same novice observers. However, when observers with different levels of experience (i.e., novices vs. a working dog expert) assessed the same dogs, agreement appeared to be strongly affected by the measurement instrument used to assess behaviour. Given that animal behaviour research often utilizes different observers with different levels of experience, our results suggest that further tests of how different observers may measure behaviour in different ways are needed across a wider variety of organisms and measurement instruments.     

TWO MODELS FOR ESTIMATING THE DISCRIMINABILITY OF FOODS AND BEVERAGES

In order to evaluate the suitability of signal detection theory methods for assessing the discriminability of foods and beverages, the discriminability of two dairy milk products that differed in fat content was measured with two detection-theoretic methods: the single-interval rating method, and the same-different method. The nominal fat contents of the milk products were 0.1 and 1.6%. Measures of discriminability for three observers were derived by fitting receiver operating characteristics (ROCs) based on equal-variance normal models to the ratings of each observer with a procedure that combined jackknifing and maximum-likelihood estimation. The fitted ROCs provided a good fit to the data indicating that the equal-variance models were appropriate for these tasks. The best-fitting estimates of d' obtained for each task were not significantly different, demonstrating that d' is a measure of sensitivity that is largely independent of the task from which it is determined. However, estimates of proportion correct obtained for each task were shown to be significantly different.     

A Statistical Analysis of Reviewer Agreement and Bias in Evaluating Medical Abstracts

Observer variability affects virtually all aspects of clinical medicine and investigation. One important aspect, not previously examined, is the selection of abstracts for presentation at national medical meetings. In the present study, 109 abstracts, submitted to the American Association for the Study of Liver Disease, were evaluated by three “blind” reviewers for originality, design-execution, importance, and overall scientific merit. Of the 77 abstracts rated for all parameters by all observers, interobserver agreement ranged between 81 and 88%. However, corresponding intraclass correlations varied between 0.16 (approaching statistical significance) and 0.37 (p < 0.01). Specific tests of systematic differences in scoring revealed statistically significant levels of observer bias on most of the abstract components. Moreover, the mean differences in interobserver ratings were quite small compared to the standard deviations of these differences. These results emphasize the importance of evaluating the simple percentage of rater agreement within the broader context of observer variability and systematic bias.     

A Double-Observer Method to Estimate Detection Rate During Aerial Waterfowl Surveys

Abstract We evaluated double-observer methods for aerial surveys as a means to adjust counts of waterfowl for incomplete detection. We conducted our study in eastern Canada and the northeast United States utilizing 3 aerial-survey crews flying 3 different types of fixed-wing aircraft. We reconciled counts of front- and rear-seat observers immediately following an observation by the rear-seat observer (i.e., on-the-fly reconciliation). We evaluated 6 a priori models containing a combination of several factors thought to influence detection probability including observer, seat position, aircraft type, and group size. We analyzed data for American black ducks (Anas rubripes) and mallards (A. platyrhynchos), which are among the most abundant duck species in this region. The best-supported model for both black ducks and mallards included observer effects. Sample sizes of black ducks were sufficient to estimate observer-specific detection rates for each crew. Estimated detection rates for black ducks were 0.62 (SE = 0.10), 0.63 (SE = 0.06), and 0.74 (SE = 0.07) for pilot-observers, 0.61 (SE = 0.08), 0.62 (SE = 0.06), and 0.81 (SE = 0.07) for other front-seat observers, and 0.43 (SE = 0.05), 0.58 (SE = 0.06), and 0.73 (SE = 0.04) for rear-seat observers. For mallards, sample sizes were adequate to generate stable maximum-likelihood estimates of observer-specific detection rates for only one aerial crew. Estimated observer-specific detection rates for that crew were 0.84 (SE = 0.04) for the pilot-observer, 0.74 (SE = 0.05) for the other front-seat observer, and 0.47 (SE = 0.03) for the rear-seat observer. Estimated observer detection rates were confounded by the position of the seat occupied by an observer, because observers did not switch seats, and by land-cover because vegetation and landform varied among crew areas. Double-observer methods with on-the-fly reconciliation, although not without challenges, offer one viable option to account for detection bias in aerial waterfowl surveys where birds are distributed at low density in remote areas that are inaccessible by ground crews. Double-observer methods, however, estimate only detection rate of animals that are potentially observable given the survey method applied. Auxiliary data and methods must be considered to estimate overall detection rate.     

Interobserver And Intraobserver Agreement In Parathyroid Scintigraphy: How Can Parathyroid Scintigraphy Be Made More Reliable?

ObjectiveTo determine the intraobserver and interobserver agreement levels in the evaluation of technetium Tc 99m sestamibi parathyroid scintigraphic images.MethodsNinety-eight patients with hyperparathyroidism were included in the study, and their parathyroid images were evaluated by 4 experienced nuclear medicine observers. The 98 cases were evaluated twice by each observer within an interval of 2 weeks. The evaluations were performed directly on workstations with use of digital images. A questionnaire was completed by each observer. The presence of a lesion, the number and the localizations of the lesions, and whether the lesion was clear or doubtful were all evaluated. Cohen kappa statistics and total agreement percentages were calculated by using SPSS version 11.0 software.ResultsThe 4 observers performed 8 different evaluations and identified a minimum of 38 and a maximum of 43 cases with a parathyroid lesion (or lesions). Both the intraobserver and the interobserver agreements were “very good” for the presence of a parathyroid lesion. The intraobserver agreement was also “very good” and the interobserver agreement was “good” (for only 1 pair of observers) or “very good” for the evaluation of the number of parathyroid lesions. The intraobserver agreement was “very good” or “good” and the interobserver agreement was “good” for the lesion localization and for the presence of a doubtful lesion.ConclusionParathyroid scintigraphy seems to be an observer independent method in the detection of a parathyroid lesion, in the determination of the number of lesions, and in the localizations of the lesions. The measured high agreement between observers increases the reliability of parathyroid scintigraphy. (Endocr Pract. 2012;18: 538-548)     

Comparaison des performances diagnostiques de la TEP au FNa avec et sans scanner couplé dans la recherche de métastases osseuses

IntroductionPerforming a CT acquisition along with a 18F-Fluoride PET study increases the effective dose delivered to the patient. We evaluated its value in the diagnosis of bone metastases.Material and methodsThree observers retrospectively analyzed 18F-Fluoride PET images (non-attenuation corrected) without CT and attenuation corrected images fused with CT in 50 patients with a cancer. The skeleton was divided into 17 areas. A score of malignancy from 1 to 5 was attributed to each area. An area-based analysis was done as well as a patient-based analysis. For the latter, the score of the area with the highest score was given to each patient. Clinical follow-up of at least 6 months, pathology or other imaging procedures served as gold standard.ResultsFor the three observers, the areas under the ROC curve of the area-based analysis were respectively 0.79, 0.97 and 0.70 without CT and 0.98, 0.98 and 0.96 with CT. The differences were significant (P < 0.05) for observers 1 and 3. The areas under the ROC curve of the patient-based analysis were respectively 0.86, 0.96 and 0.85 without CT and 0.97, 0.98 and 0.99 with CT and there was a significant difference for observer 3.ConclusionPerforming a CT acquisition along with an 18F-Fluoride PET study increases the homogeneity of the results obtained by different observers.     

Strong observer effect on tree microhabitats inventories: A case study in a French lowland forest

Validating biodiversity indicators requires an analysis of their applicability, their range of validity and their degree of correlation with the biodiversity they are supposed to represent. In this process, assessing the magnitude of observer effect is an essential step, especially if non-specialist observers are involved. Tree microhabitats – woodpecker cavities, cracks and bark characteristics – are reputed to be easily detected by non-specialists as microhabitat observation does not require prior forestry or ecology knowledge. We therefore quantified the probabilities of true and false positive detections made by observers during inventories.Within two 0.5 ha plots in a forest reserve that has not been harvested for at least 150 years, 14 observers with various backgrounds visually inventoried microhabitats on 106 oak (Quercus petraea and Quercus robur) and beech (Fagus sylvatica) trees. We used parametric and Bayesian statistics to compare these observers’ recorded observations with results from an independent census.The mean number of microhabitats per tree varied widely among observers – from 1.4 to over 3. Only five observers reported a mean number of microhabitats per tree that was statistically equivalent to the reference census. The probability of true detection also varied among observers for each microhabitat (from to 0 to 1) as did the probability of false positive detection (from 0 to 0.7). These results show that microhabitat inventories are particularly prone to observer effects.Such strong observer effects weaken the usefulness of microhabitats as biodiversity indicators. If microhabitat inventories are to be developed, we recommend controlling for observer effects by (i) defining standard operating procedures and multiplying the number of observer training sessions and of consensual standardization censuses; (ii) using pairs of observers to record microhabitats whenever possible (though the efficiency of this method remains to be tested); (iii) planning fieldwork so that the factors of interest are not confused with observer effects; and (iv) integrating observer profiles into the statistical models used to analyze the data.     

Observer variation in field assessments of vegetation condition: Implications for biodiversity conservation

Summary   Uncertainty in assessments of vegetation condition that are used to inform land management and planning decisions for biodiversity conservation in Australia may lead to unexpected outcomes, including loss of biodiversity. This study investigates observer error in field estimates of vegetation attributes, one component of uncertainty in assessments of vegetation condition. Ten observers conducted vegetation condition assessments using two assessment protocols (BioMetric and Habitat Hectares) on 20 sites in a grassy woodland community. Observers' estimates varied substantially across multiple scoring categories for all vegetation attributes on almost all sites. Across all sites, the average coefficient of variation in total vegetation condition scores was 15–18% for both protocols, with a maximum of 60%. The primary cause of variation in total vegetation condition scores was random error in raw estimates of vegetation attributes, although sensitivity of some highly weighted attributes to error exacerbated variation in some cases. Observers generally agreed on the total scores and ranks of highly degraded (pasture) sites, but were less consistent on other sites. Rank correlations between pairs of observers were stronger for Habitat Hectares, suggesting BioMetric may be slightly more sensitive to observer error. It is recommended that: (i) research is undertaken into methods for reducing observer error; (ii) review is made of the sensitivity of index scoring structures to observer error; (iii) field observers estimate uncertainty around point estimates of vegetation condition; and, (iv) decision-makers explicitly incorporate uncertainty into the decision-making processes and aim for outcomes that are robust to this uncertainty.     

Personality in Captive Chimpanzees: Use of a Novel Rating Procedure

We present a new method to assess the personalities of captive chimpanzees (Pan troglodytes). Using individually generated bipolar constructs, experienced observers rated the personalities of chimpanzees on a five-point scale. Each observer chose a subset of his/her own constructs and we performed a separate factor analysis on each set of ratings. The analyses indicate four factors: dominance, sociability, machiavellianism, and anxiety. We propose that this factor structure will be useful in the development of a single rating instrument with which to assess the personalities of captive chimpanzees.     

Determining Observer Reliability in Counts of River Otter Tracks

ABSTRACT In many research projects, reliability of collected data is dependent on reliability of field observers. However, it is uncommon for observer reliability to be either measured or reported in wildlife research. We tested whether observer skill affected outcomes of a northern river otter (Lontra canadensis) track survey conducted by the Texas Parks and Wildlife Department. Observers recorded presence of tracks at bridge sites (n = 250) throughout a 27-county region in east Texas, USA. Logistic regression indicated that observers were significantly associated with frequency of reported otter tracks. Because observers were not assigned to bridges at random, we tested and found associations between the bridges surveyed by each observer (SURVEY ROUTE) and habitat variables (WATERSHED, VEGETATION-TYPE, WATER-TYPE, BRIDGE-AREA) that may have influenced otter presence and probability of detection. A standardized tracker evaluation procedure indicated that experienced observers (n = 7) misidentified 37% of otter tracks. Additionally, 26% of tracks from species determined to be “otter-like” were misidentified as otter tracks. We recommend that observer skill in identification of animal tracks and other indirect signs be measured to detect and reduce observer errors in wildlife monitoring.     

Behavioral differences as predictors of breeding status in captive cheetahs

Individual behavioral variation of 44 adult captive‐born cheetahs, Acinonyx jubatus, was assessed using observer and keeper ratings on a variety of behavioral attributes. Inter‐rater consensus was high for most questionnaire items, several of which were significantly correlated with direct behavioral measures recorded in a mirror‐image experiment. Principal component analysis was applied to identify patterns of individual variation as measured by observer and keeper ratings. Three major components, labeled as tense‐fearful, excitable‐vocal, and aggressive, accounted for 69% of the observed variation. Females showed significantly higher scores on the component tense‐fearful than males. Non‐breeders of both sexes scored significantly higher on the component tense‐fearful than breeders. Assessment of individual behavioral variation through questionnaire ratings may therefore provide a simple and non‐invasive tool for predicting an individual's ability to adjust to the constraints of certain husbandry regimens and to reproduce in captivity. Simple measures of behavioral attributes may offer new insights for solving breeding problems and improving conservation management of endangered species in captivity. Zoo Biol 18:335–349, 1999. © 1999 Wiley‐Liss, Inc.     

An examination of observer bias as a source of error in surveys of seagrass shoots

Abstract Mensurative experiments investigated the effects of different observers on estimates of the density of shoots of two species of seagrass: Posidonia australis Hook and Zostera capricorni Aschers. Balanced programmes of sampling were used to examine variation in counts of seagrass shoots attributable to different observers, sizes of quadrats, depths and locations within large beds of each species of seagrass. A separate experiment examined differences between novice observers and a more experienced observer, when an ‘optimal’ size of sampling unit was used. Estimated densities of Zostera shoots varied inconsistently among observers, quadrats, depths and locations. Differences between observers were not affected by the size of quadrat used to count Posidonia shoots, but varied between locations in the seagrass bed. Experience had only a minor impact on biases. Only two of 12 novices produced counts that were different from the experienced observer. These results emphasize the importance of considering both accuracy and precision in the design of field studies of seagrasses.     

Testing the importance of auditory detections in avian point counts

ABSTRACT.   Recent advances in the methods used to estimate detection probability during point counts suggest that the detection process is shaped by the types of cues available to observers. For example, models of the detection process based on distance-sampling or time-of-detection methods may yield different results for auditory versus visual cues because of differences in the factors that affect the transmission of these cues from a bird to an observer or differences in an observer's ability to localize cues. Previous studies suggest that auditory detections predominate in forested habitats, but it is not clear how often observers hear birds prior to detecting them visually. We hypothesized that auditory cues might be even more important than previously reported, so we conducted an experiment in a forested habitat in North Carolina that allowed us to better separate auditory and visual detections. Three teams of three observers each performed simultaneous 3-min unlimited-radius point counts at 30 points in a mixed-hardwood forest. One team member could see, but not hear birds, one could hear, but not see, and the third was nonhandicapped. Of the total number of birds detected, 2.9% were detected by deafened observers, 75.1% by blinded observers, and 78.2% by nonhandicapped observers. Detections by blinded and nonhandicapped observers were the same only 54% of the time. Our results suggest that the detection of birds in forest habitats is almost entirely by auditory cues. Because many factors affect the probability that observers will detect auditory cues, the accuracy and precision of avian point count estimates are likely lower than assumed by most field ornithologists.     

Individual differences in responses of Norway rats to social induction of food preferences

Each of 36 observer rats was: (1) exposed to a demonstrator rat that had eaten an unpalatable, cayenne-pepper-flavored diet (Diet Cay), then tested to determine its willingness to eat Diet Cay and (2) exposed to a demonstrator rat that had eaten a palatable diet (Diet NPT) to which the observer had previously learned an aversion, then tested to determine its willingness to eat Diet NPT. In both instances, some observers ate substantial amounts of the diet that their respective demonstrators had eaten, while other observers did not. No consistency was found across the two situations in the relative susceptibility of individual observer rats to social influences on their food choices. In a second experiment, observer rats interacted, at 3 day intervals, with demonstrator rats that had each eaten different diets. After each interaction, all observers were given a choice test to determine their preferences for the diet that their demonstrators had eaten. Again, there was no consistency in the relative strength of individual observer rats' socially induced preferences for diets fed to demonstrators. Stable individual differences in magnitude of susceptibility to social influence on food preference did not account for a detectable proportion of observed variance in diet selection.     

Effects of the observation method (direct v. from video) and of the presence of an observer on behavioural results in veal calves

This study aimed at assessing the effect of the observation method (direct or from video) and the effect of the presence of an observer on the behavioural results in veal calves kept on a commercial farm. To evaluate the effect of the observation method, 20 pens (four to five calves per pen) were observed by an observer for 60 min (two observation sessions of 30 min) and video-recorded at the same time. To evaluate the effect of the presence of the observer in front of the pen, 24 pens were video-recorded on 4 consecutive days and an observer was present in front of each pen for 60 min (two observation sessions of 30 min) on the third day. Behaviour was recorded using instantaneous scan sampling. For the study of the observer's effect, the analysis was limited to the posture, abnormal oral behaviour and manipulation of substrates. The two observation methods gave similar results for the time spent standing, but different results for all other behaviours. The presence of an observer did not affect the behaviour of calves at day level; however, their behaviour was affected when the observer was actually present in front of the pens. A higher percentage of calves were standing and were manipulating substrate in the presence of the observer, but there was no effect on abnormal oral behaviour. In conclusion, direct observations are a more suitable observation method than observations from video recordings for detailed behaviours in veal calves. The presence of an observer has a short-term effect on certain behaviours of calves that will have to be taken into consideration when monitoring these behaviours.     

OBSERVER VARIABILITY IN PINNIPED COUNTS: GROUND-BASED ENUMERATION OF WALRUSES AT HAUL-OUT SITES

Pinnipeds are often monitored by counting individuals at haul-out sites, but the often large numbers of densely packed individuals at these sites are difficult to enumerate accurately. Errors in enumeration can induce bias and reduce precision in estimates of population size and trend. We used data from paired observers monitoring walrus haul-outs in Bristol Bay, Alaska, to quantify observer variability and assess its relative importance. The probability of a pair of observers making identical counts was < 0.1 for walrus groups with >50 individuals. Mean count differences ranged up to 25% for the largest counts, depending on beach and observers. In at least some cases, there was a clear tendency for counts of one observer to be consistently greater than counts of the other observer in a pair, indicating that counts of at least one of the observers were biased. These results suggest that efforts to improve accuracy of counts will be worthwhile. However, we also found that variation among observers was relatively small compared to variation among visits to a beach so that efforts to account for other sources of variation will be more important.