Models and estimators linking individual-based and sample-based rarefaction,extrapolation and comparison of assemblages

Aims In ecology and conservation biology, the number of species counted in a biodiversity study is a key metric but is usually a biased underestimate of total species richness because many rare species are not detected. Moreover, comparing species richness among sites or samples is a statistical challenge because the observed number of species is sensitive to the number of individuals counted or the area sampled. For individual-based data, we treat a single, empirical sample of species abundances from an investigator-defined species assemblage or community as a reference point for two estimation objectives under two sampling models: estimating the expected number of species (and its unconditional variance) in a random sample of (i) a smaller number of individuals (multinomial model) or a smaller area sampled (Poisson model) and (ii) a larger number of individuals or a larger area sampled. For sample-based incidence (presence–absence) data, under a Bernoulli product model, we treat a single set of species incidence frequencies as the reference point to estimate richness for smaller and larger numbers of sampling units.Methods The first objective is a problem in interpolation that we address with classical rarefaction (multinomial model) and Coleman rarefaction (Poisson model) for individual-based data and with sample-based rarefaction (Bernoulli product model) for incidence frequencies. The second is a problem in extrapolation that we address with sampling-theoretic predictors for the number of species in a larger sample (multinomial model), a larger area (Poisson model) or a larger number of sampling units (Bernoulli product model), based on an estimate of asymptotic species richness. Although published methods exist for many of these objectives, we bring them together here with some new estimators under a unified statistical and notational framework. This novel integration of mathematically distinct approaches allowed us to link interpolated (rarefaction) curves and extrapolated curves to plot a unified species accumulation curve for empirical examples. We provide new, unconditional variance estimators for classical, individual-based rarefaction and for Coleman rarefaction, long missing from the toolkit of biodiversity measurement. We illustrate these methods with datasets for tropical beetles, tropical trees and tropical ants.Important findings Surprisingly, for all datasets we examined, the interpolation (rarefaction) curve and the extrapolation curve meet smoothly at the reference sample, yielding a single curve. Moreover, curves representing 95% confidence intervals for interpolated and extrapolated richness estimates also meet smoothly, allowing rigorous statistical comparison of samples not only for rarefaction but also for extrapolated richness values. The confidence intervals widen as the extrapolation moves further beyond the reference sample, but the method gives reasonable results for extrapolations up to about double or triple the original abundance or area of the reference sample. We found that the multinomial and Poisson models produced indistinguishable results, in units of estimated species, for all estimators and datasets. For sample-based abundance data, which allows the comparison of all three models, the Bernoulli product model generally yields lower richness estimates for rarefied data than either the multinomial or the Poisson models because of the ubiquity of non-random spatial distributions in nature.     

Quantifying biodiversity: procedures and pitfalls in the measurement and comparison of species richness

Species richness is a fundamental measurement of community and regional diversity, and it underlies many ecological models and conservation strategies. In spite of its importance, ecologists have not always appreciated the effects of abundance and sampling effort on richness measures and comparisons. We survey a series of common pitfalls in quantifying and comparing taxon richness. These pitfalls can be largely avoided by using accumulation and rarefaction curves, which may be based on either individuals or samples. These taxon sampling curves contain the basic information for valid richness comparisons, including category–subcategory ratios (species-to-genus and species-to-individual ratios). Rarefaction methods – both sample-based and individual-based – allow for meaningful standardization and comparison of datasets. Standardizing data sets by area or sampling effort may produce very different results compared to standardizing by number of individuals collected, and it is not always clear which measure of diversity is more appropriate. Asymptotic richness estimators provide lower-bound estimates for taxon-rich groups such as tropical arthropods, in which observed richness rarely reaches an asymptote, despite intensive sampling. Recent examples of diversity studies of tropical trees, stream invertebrates, and herbaceous plants emphasize the importance of carefully quantifying species richness using taxon sampling curves.     

Additive partitioning of rarefaction curves and species-area relationships: unifying alpha-, beta- and gamma-diversity with sample size and habitat area

Additive partitioning of species diversity is widely applicable to different kinds of sampling regimes at multiple spatial and temporal scales. In additive partitioning, the diversity within and among samples ( α and β ) is expressed in the same units of species richness, thus allowing direct comparison of α and β . Despite its broad applicability, there are few demonstrated linkages between additive partitioning and other approaches to analysing diversity. Here, we establish several connections between diversity partitions and patterns of habitat occupancy, rarefaction, and species–area relationships. We show that observed partitions of species richness are equivalent to sample-based rarefaction curves, and expected partitions from randomization tests are approximately equivalent to individual-based rarefaction. Additive partitions can also be applied to species–area relationships to determine the relative contributions of factors influencing the β -diversity among habitat fragments.     

Evaluating detectability of freshwater fish assemblages in tropical streams: Is hand-seining sufficient?

Unprecedented threats to natural ecosystems mean that accurate quantification of biodiversity is a priority, particularly in the tropics which are underrepresented in monitoring schemes. Data from a freshwater fish assemblage in Trinidad were used to evaluate the effectiveness of hand-seining as a survey method in tropical streams. We uncovered large differences in species detectability when hand-seining was used alone, in comparison with when hand-seining and electrofishing were used together. The addition of electrofishing increased the number of individuals caught threefold, and increased the biomass fivefold. Some species were never detected using hand-seining, resulting in significant underestimates of species richness; rarefaction curves suggest that even when hand-seining effort increases, species richness is still underestimated. Diversity indices (Shannon and Simpson index) reveal that diversity was also significantly lower for hand-seined samples. Furthermore, the results of multivariate analyses investigating assemblage structure also differed significantly depending on whether they were based on hand-seined data alone, or a combination of hand-seining and electrofishing. Despite the extra equipment and maintenance required, these findings underline the value of including electrofishing when sampling tropical freshwater streams.     

A sampling optimization analysis of soil‐bugs diversity (Crustacea,Isopoda, Oniscidea)

Biological diversity analysis is among the most informative approaches to describe communities and regional species compositions. Soil ecosystems include large numbers of invertebrates, among which soil bugs (Crustacea, Isopoda, Oniscidea) play significant ecological roles. The aim of this study was to provide advices to optimize the sampling effort, to efficiently monitor the diversity of this taxon, to analyze its seasonal patterns of species composition, and ultimately to understand better the coexistence of so many species over a relatively small area. Terrestrial isopods were collected at the Natural Reserve “Saline di Trapani e Paceco” (Italy), using pitfall traps monthly monitored over 2 years. We analyzed parameters of α‐ and β‐diversity and calculated a number of indexes and measures to disentangle diversity patterns. We also used various approaches to analyze changes in biodiversity over time, such as distributions of species abundances and accumulation and rarefaction curves. As concerns species richness and total abundance of individuals, spring resulted the best season to monitor Isopoda, to reduce sampling efforts, and to save resources without losing information, while in both years abundances were maximum between summer and autumn. This suggests that evaluations of β‐diversity are maximized if samples are first collected during the spring and then between summer and autumn. Sampling during these coupled seasons allows to collect a number of species close to the γ‐diversity (24 species) of the area. Finally, our results show that seasonal shifts in community composition (i.e., dynamic fluctuations in species abundances during the four seasons) may minimize competitive interactions, contribute to stabilize total abundances, and allow the coexistence of phylogenetically close species within the ecosystem.     

Incorporating functional dissimilarities into sample‐based rarefaction curves: from taxon resampling to functional resampling

Question: Indices of functional diversity have been seen as the key for integrating information on species richness with measures that focus on those components of community composition related to ecosystem functioning. For comparing species richness among habitats on an equal‐effort basis, so‐called sample‐based rarefaction curves may be used. Given a study area that is sampled for species presence and absence in N plots, sample‐based rarefaction generates the expected number of accumulated species as the number of sampled plots increases from 1 to N. Accordingly, the question for this study is: can we construct a ‘functional rarefaction curve’ that summarizes the expected functional dissimilarity between species when n plots are drawn at random from a larger pool of N plots? Methods: In this paper, we propose a parametric measure of functional diversity that is obtained by combining sample‐based rarefaction techniques that are usually applied to species richness with Rao's quadratic diversity. For a given set of N presence/absence plots, the resulting measure summarizes the expected functional dissimilarity at an increasingly larger cumulative number of plots n (n≤N). Results and Conclusions: Due to its parametric nature, the proposed measure is progressively more sensitive to rare species with increasing plot number, thus rendering this measure adequate for comparing the functional diversity of species assemblages that have been sampled with variable effort.     

Patchiness of local species richness and its implication for large-scale diversity patterns: an example from the middle Miocene of the Paratethys

The best hope for understanding global diversity patterns is to compare local assemblages, which are mostly preserved in taphonomically-complex shell beds. The present study investigates the variability in faunal composition and diversity at the scale of a single outcrop. A total of 152 species (3315 shells) occurred in 25 samples from 5 tempestitic shell beds. Although sampling intensity was high, total species richness was not captured by far at the hierarchical levels present (outcrop, shell beds, samples) because the majority of species is rare. In contrast, sampling intensity was sufficient to cover the most abundant species, as indicated by stable evenness values. Four taxa dominate the assemblage, but their rank order differs strongly between individual shell beds and individual samples; significant differences between some shell beds are evident for faunal composition, and one shell bed differs from all others with respect to species accumulation curves. Within shell beds, rarefaction curves are generally characterized by strongly overlapping confidence intervals, but outliers occur in three of five shell beds. Patchiness is additionally indicated by a wide scatter of diversity indices in some shell beds and by a wide scatter of samples of one shell bed in an ordination on faunal composition. Most of the outcrop-scale variability in faunal composition and diversity can be related to differences between shell beds. This suggests that sampling a single shell bed of the outcrop is insufficient to characterize the local fauna and its diversity, even when sampling intensity (i.e. the number of samples and shells) within the shell bed was high. Similarly, a single sample from such a shell bed may not be sufficient to characterize its diversity, even when the number of counted shells was high. It is therefore confirmed that sampling strategy and sampling intensity are crucial to confidently characterize the shelly assemblages at such a small spatial scale and that dispersed sampling effort with many small replicate samples will characterize a local assemblage and its diversity better than a few large samples. Diversity comparisons of individual samples between localities must account for the high variability present at the smaller spatial scale, as observed in our study.     

A unified mathematical framework for the measurement of richness and evenness within and among multiple communities

Biodiversity can be divided into two aspects: richness (the number of species or other taxa in a community or sample) and evenness (a measure of the distribution of relative abundances of different taxa in a community or sample). Sample richness is typically evaluated using rarefaction, which normalizes for sample size. Evenness is typically summarized in a single value. It is shown here that Hurlbert's probability of interspecific encounter (Δ₁), a commonly used sample-size independent measure of evenness, equals the slope of the steepest part of the rising limb of a rarefaction curve. This means that rarefaction curves provide information on both aspects of diversity. In addition, regional diversity (gamma) can be broken down into the diversity within local communities (alpha) and differences in taxonomic composition among local communities (beta). Beta richness is expressed by the difference between the composite rarefaction curve of all samples in a region with the collector's curve for the same samples. The differences of the initial slopes of these two curves reflect the beta evenness thanks to the relationship between rarefaction and Δ₁. This relationship can be further extended to help interpret species-area curves (SAC's). As previous authors have described, rarefaction provides the null hypothesis of passive sampling for SAC's, which can be interpreted as regional collector's curves. This allows evaluation of richness and evenness at local and regional scales using a single family of well-established, mathematically related techniques.     

Functional rarefaction: estimating functional diversity from field data

Studies in biodiversity-ecosystem function and conservation biology have led to the development of diversity indices that take species' functional differences into account. We identify two broad classes of indices: those that monotonically increase with species richness (MSR indices) and those that weight the contribution of each species by abundance or occurrence (weighted indices). We argue that weighted indices are easier to estimate without bias but tend to ignore information provided by rare species. Conversely, MSR indices fully incorporate information provided by rare species but are nearly always underestimated when communities are not exhaustively surveyed. This is because of the well-studied fact that additional sampling of a community may reveal previously undiscovered species. We use the rarefaction technique from species richness studies to address sample-size-induced bias when estimating functional diversity indices. Rarefaction transforms any given MSR index into a family of unbiased weighted indices, each with a different level of sensitivity to rare species. Thus rarefaction simultaneously solves the problem of bias and the problem of sensitivity to rare species. We present formulae and algorithms for conducting a functional rarefaction analysis of the two most widely cited MSR indices: functional attribute diversity (FAD) and Petchey and Gaston's functional diversity (FD). These formulae also demonstrate a relationship between three seemingly unrelated functional diversity indices: FAD, FD and Rao's quadratic entropy. Statistical theory is also provided in order to prove that all desirable statistical properties of species richness rarefaction are preserved for functional rarefaction.     

Aspects of Diversity Measurement for Microbial Communities

A useful measure of diversity was calculated for microbial communities collected from lake water and sediment samples using the Shannon index (H′) and rarefaction [E(S)]. Isolates were clustered by a numerical taxonomy approach in which limited (<20) tests were used so that the groups obtained represented a level of resolution other than species. The numerical value of diversity for each sample was affected by the number of tests used; however, the relative diversity compared among several sampling locations was the same whether 11 or 19 characters were examined. The number of isolates (i.e., sample size) strongly influenced the value of H′ so that unequal sized samples could not be compared. Rarefaction accounts for differences in sample size inherently so that such comparisons are made simple. Due to the type of sampling carried out by microbiologists, H′ is estimated and not determined and therefore requires a statement of error associated with it. Failure to report error provided potentially misleading results. Calculation of the variance of H′ is not a simple matter and may be impossible when handling a large number of samples. With rarefaction, the variance of E(S) is readily determined, facilitating the comparison of many samples.     

Deriving site-specific species pools from large databases

Defining the species pool of a community is crucial for many types of ecological analyses, providing a foundation to metacommunity, null modelling or dark diversity frameworks. It is a challenge to derive the species pool empirically from large and heterogeneous databases. Here, we propose a method to define a site-specific species pool (SSSP), i.e. the probabilistic set of species that may co-occur with the species of a target community. Using large databases with geo-referenced records that comprise full plant community surveys, our approach characterizes each site by its own species pool without requiring a pre-defined habitat classification. We calculate the probabilities of each species in the database to occur in the target community using Beals’ index of sociological favourability, and then build sample-based rarefaction curves from neighbouring records with similar species composition to estimate the asymptotic species pool size. A corresponding number of species is then selected among the species having the highest occurrence probability, thus defining both size and composition of the species pool. We tested the robustness of this approach by comparing SSSPs obtained with different spatial extents and dissimilarity thresholds, fitting different models to the rarefaction curves, and comparing the results obtained when using Beals co-occurrence probabilities or presence/absence data. As an example application, we calculated the SSSPs for all calcareous grassland records in the German Vegetation Reference Database, and show how our method could be used to 1) produce grain-dependent estimations of species richness across plots, 2) derive scalable maps of species richness and 3) define the full list of species composing the SSSP of each target site. By deriving the species pool exclusively from community characteristics, the SSSP framework presented here provides a robust approach to bridge biodiversity estimations across spatial scales.     

When species accumulation curves intersect: implications for ranking diversity using small samples

Diversity in biological communities frequently is compared using species accumulation curves, plotting observed species richness versus sample size. When species accumulation curves intersect, the ranking of communities by observed species richness depends on sample size, creating inconsistency in comparisons of diversity. We show that species accumulation curves for two communities are expected to intersect when the community with lower actual species richness has higher Simpson diversity (probability that two random individuals belong to different species). This may often occur when comparing communities that differ in habitat heterogeneity or disturbance, as we illustrate using data from neotropical butterflies. In contrast to observed species richness, estimated Simpson diversity always produces a consistent expected ranking among communities across sample sizes, with the statistical accuracy to confidently rank communities using small samples. Simpson diversity should therefore be particularly useful in rapid assessments to prioritize areas for conservation.     

Tree Diversity in Western Kenya: Using Profiles to Characterise Richness and Evenness

Species diversity is a function of the number of species and the evenness in the abundance of the component species. We calculated diversity and evenness profiles, which allowed comparing the diversity and evenness of communities. We applied the methodology to investigate differences in diversity among the main functions of trees on western Kenyan farms. Many use-groups (all trees and species that provide a specific use) could not be ranked in diversity or evenness. No use-group had perfectly even distributions. Evenness could especially be enhanced for construction materials, fruit, ornamental, firewood, timber and medicine, which included some of the most species-rich groups of the investigated landscape. When considering only the evenness in the distribution of the dominant species, timber, medicine, fruit and beverage ranked lowest (> 60% of trees belonged to the dominant species of these groups). These are also use-groups that are mainly grown by farmers to provide cash through sales. Since not all communities can be ranked in diversity, studies that attempt to order communities in diversity should not base the ordering on a single index, or even a combination of several indices, but use techniques developed for diversity ordering such as the Rényi diversity profile. The rarefaction of diversity profiles described in this article could be used in studies that compare results from surveys with different sample sizes.     

Mapping patterns of ferns species richness through the use of herbarium data

This paper aims to analyse the spatial patterns of sampling effort and species richness of pteridophyte in a well-investigated region as Tuscany, Italy, by using data stored from a geodatabase storing information on the specimens preserved in the main herbaria of the region. A total of 6,905 records about pteridophyte specimens were extracted from the geodatabase, and 5,638 of such specimens were studied through the use of spatial statistical techniques. The data about the sampling effort and species richness were analysed in relation to topographical variables to assess any significant relationship. Specimen-based rarefaction techniques were used to compare areas with different number of detected species. The analysis of the sampling effort data showed a nonhomogeneous distribution of herbarium data, with some areas being intensively sampled and others being almost unsampled. Thus, the geographical distribution of specimens was extremely clustered. The comparison across geographical areas through specimen-based rarefaction curves showed great differences in species richness and sampling completeness. The analysis of the residuals of species–area relationships evidenced that the distance to water bodies was the only significant topographical variable in controlling species diversity.     

A conceptual guide to measuring species diversity

Three metrics of species diversity – species richness, the Shannon index and the Simpson index – are still widely used in ecology, despite decades of valid critiques leveled against them. Developing a robust diversity metric has been challenging because, unlike many variables ecologists measure, the diversity of a community often cannot be estimated in an unbiased way based on a random sample from that community. Over the past decade, ecologists have begun to incorporate two important tools for estimating diversity: coverage and Hill diversity. Coverage is a method for equalizing samples that is, on theoretical grounds, preferable to other commonly used methods such as equal-effort sampling, or rarefying datasets to equal sample size. Hill diversity comprises a spectrum of diversity metrics and is based on three key insights. First, species richness and variants of the Shannon and Simpson indices are all special cases of one general equation. Second, richness, Shannon and Simpson can be expressed on the same scale and in units of species. Third, there is no way to eliminate the effect of relative abundance from estimates of any of these diversity metrics, including species richness. Rather, a researcher must choose the relative sensitivity of the metric towards rare and common species, a concept which we describe as ‘leverage.' In this paper we explain coverage and Hill diversity, provide guidelines for how to use them together to measure species diversity, and demonstrate their use with examples from our own data. We show why researchers will obtain more robust results when they estimate the Hill diversity of equal-coverage samples, rather than using other methods such as equal-effort sampling or traditional sample rarefaction.     

Are there habitats that contribute best to plant species diversity in coastal dunes?

The following paper describes patterns of diversity across major habitat types in a relatively well preserved coastal dune system in central Italy. The research addresses the following questions: (a) whether different habitats defined on the base of a land cover map support similar levels of biodiversity in terms of vascular flora richness and number of rare and endangered species, and (b) how each habitat contributes to the total species diversity of the coastal environment. A random stratified sampling approach based on a detailed land cover map was applied to construct rarefaction curves for each habitat type and to estimate total species richness. In addition, the number of exclusive, rare and endangered species was calculated for each habitat type. Results highlight the importance of the coastal dune zonation (embryo-dune, main dune, transition and stabilized dune) in species conservation because they harbour progressively higher species richness. However, differences among these habitats were not significant, so no particular species rich “hotspots” could be evidenced. On the contrary, rarefaction curves show that the upper beach (strand) habitat sustains significantly smaller number of species, but surprisingly, it shows the highest rarity values and highest proportion of endangered species. Therefore, for the establishment of successful biodiversity conservation programs in these coastal environments, it is imperative not only to conserve biologically rich hotspots but also to include species poor habitats containing endangered or unique elements. Thus, the complete coastal vegetation mosaic including all coastal habitats is important to adequately characterize the plant species diversity of coastal dune ecosystems.     

hp‐rare 1.0: a computer program for performing rarefaction on measures of allelic richness

The number of alleles in a sample (allelic richness) is a fundamental measure of genetic diversity. However, this diversity measure has been difficult to use because large samples are expected to contain more alleles than small samples. The statistical technique of rarefaction compensates for this sampling disparity. Here I introduce a computer program that performs rarefaction on private alleles and hierarchical sampling designs.     

Assessing the diversity pattern of cryophilous plant species in high elevation habitats

This study aimed to better document the diversity and distribution patterns of vascular cryophilous species across major habitat types in a high-elevation Mediterranean system in central Italy. The research addressed the following questions: (a) whether different habitats support similar levels of biodiversity in terms of total vascular plants richness and cryophilous species richness, and (b) how each habitat contributes to the total cryophilous species diversity. A random stratified sampling approach based on a habitat map was applied to construct rarefaction curves for overall cryophilous species richness and habitat type-specific cryophilous richness. Rarefaction curves were also constructed for all-species and exclusive species. To determine whether the targeted species represented a constant proportion of all species, the ratio between the rarefaction curves of the cryophilous species and all species was also calculated. The results highlight the importance of the different habitat types in overall and cryophilous species conservation because these different habitat types had progressively higher richness values. At the regional scale, steep slopes had the highest species diversity, the greatest exclusive species richness and a steep rarefaction curve. The diversity pattern of cryophilous taxa was not related to the general pattern of total species richness, with these species being more common in three habitat types with extreme environmental conditions: ridges, cliffs, and screes. For the establishment of successful biodiversity conservation programs, it is imperative to include species-poor habitats containing a high proportion of cryophilous species, which are considered to be threatened by climate warming.     

The effects of protection from fishing on species richness: distinguishing between alternative explanations

Marine reserves that prohibit fishing often result in greater densities of individuals and more species than adjacent fished areas. However, simple conclusions about their effects on species richness are confounded, because more species are expected to occur wherever there are more individuals. Here, there is an important distinction between the number of species per sampling unit (species density), and species richness measured as the number of species per given number of individuals. When conservation of species richness is an important goal, analyses need to discriminate between the alternative explanations for differences in the number of species. We used rarefaction to test whether species richness was higher in two ‘no-take’ marine reserves after controlling for differences in the density of individuals. We surveyed each reserve in three different years. There was a higher density of individuals and species in each reserve than in adjacent fished areas. However, rarefaction analyses indicated that effects on species richness were weak after controlling for the number of individuals: slightly higher species richness was recorded inside each reserve in one of three surveys, but the difference was small, and was apparent only when the maximum number of individuals was approached. Our results therefore indicate that patterns in species density were not reflected by patterns in species richness—the application of rarefaction methods is needed to determine the responses of species richness to protection elsewhere. The distinction between species density and species richness will not be important in all situations, but when it is important, inferences about species richness cannot be reliably deduced from measurements of species density.