Intercalibration and comparison – major results and conclusions from the STAR project

The main results of the STAR project on the intercalibration of boundaries of European assessment systems and comparison between assessment methods are summarized here. The main findings are outlined in the context of the Water Framework Directive that requires reliable instructions to be set up on how to use and harmonise assessment systems and methods for European rivers. The main papers published on these subjects by STAR partners are reviewed, with focus on major questions addressed and approaches used for investigation. The need for broad coverage of geographic ranges and pressure gradients, together with the goal of providing outcomes appropriate to the effective application of the WFD are emphasized. Extensive datasets from a wide range of countries, stream types and sites and a large number of methods, metrics and approaches are compared and tested and various cross-cutting themes emerged. Among these, the value of the use of benchmarking systems for comparison and intercalibration is highlighted. Two ways of looking for comparability of assessment systems results were analyzed: a) by adopting identical sampling techniques across Europe and b) by harmonizing the classification results of the national assessment systems. In addition, the need, in the intercalibration process, for a proper definition of the criteria for reference conditions is underlined. This is because their imprecision now represents one of the major weaknesses of the whole intercalibration process. Direct and indirect approaches to intercalibration are considered and commented on for their potential use in distinct circumstances. Finally, the use common metrics for the intercalibration process, which make comparability across Europe valid, is tested and indeed encouraged.     

Bringing European river quality into line: an exercise to intercalibrate macro-invertebrate classification methods

The Water Framework Directive (WFD) requires intercalibration to be performed to ensure that ecological status, as defined by the boundary values of national biological assessment systems, is consistent with the definitions outlined in the WFD and comparable between Member States (MS). This article describes an intercalibration of 17 national river macro-invertebrate assessment methods from the Central and Baltic regions of Europe. We explore the hypothesis that intercalibration should be successful if ratios of the observed biota to that expected in reference condition are used to compare assessments of different national assessment systems. National boundaries expressed as ecological quality ratios (EQRs) were converted to values of a common multi-metric for the purpose of comparison. Twelve MS for the High/Good boundary and nine MS for the Good/Moderate boundary (and four MS who subsequently harmonised their boundaries) were within ±0.05 EQR units of the intercalibration boundaries and were deemed to be of comparable ecological standard. The use of a reference-based approach was deemed to be successful given that all the critical pre-requisites for intercalibration were satisfied. The boundaries derived from this intercalibration represent the first common interpretation of the ecological status of rivers based on macro-invertebrate assessment methods across Europe.     

Ecological status assessment of European lakes: a comparison of metrics for phytoplankton, macrophytes, benthic invertebrates and fish

Data on phytoplankton, macrophytes, benthic invertebrates and fish from more than 2000 lakes in 22 European countries were used to develop and test metrics for assessing the ecological status of European lakes as required by the Water Framework Directive. The strongest and most sensitive of the 11 metrics responding to eutrophication pressure were phytoplankton chlorophyll a, a taxonomic composition trophic index and a functional traits index, the macrophyte intercalibration taxonomic composition metric and a Nordic lake fish index. Intermediate response was found for a cyanobacterial bloom intensity index (Cyano), the Ellenberg macrophyte index and a multimetric index for benthic invertebrates. The latter also responded to hydromorphological pressure. The metrics provide information on primary and secondary impacts of eutrophication in the pelagic and the littoral zone of lakes. Several of these metrics were used as common metrics in the intercalibration of national assessment systems or have been incorporated directly into the national systems. New biological metrics have been developed to assess hydromorphological pressures, based on aquatic macrophyte responses to water level fluctuations, and on macroinvertebrate responses to morphological modifications of lake shorelines. These metrics thus enable the quantification of biological impacts of hydromorphological pressures in lakes.     

Harmonising the bioassessment of large rivers in the absence of near‐natural reference conditions – a case study of the Danube River

1. International river catchments pose challenges for effective water resource management. Catchment‐wide strategies are often complicated by differences in national bioassessment and quality classification. Intercalibration efforts aim to harmonise these differences, but rely on the consistent delineation of near‐natural reference sites that are almost unavailable in today’s landscape, especially for large rivers. 2. We introduce the concept of alternative benchmarking that is based on the notion of aquatic communities at similar (low) levels of impairment associated with least‐disturbed conditions (LDC) as defined by abiotic criteria. Using data acquired during the second Joint Danube Survey, we defined LDC sites based on a multivariate gradient of anthropogenic pressures, mostly related to morphological deterioration, that spans the entire navigable Danube. 3. The river was subdivided into four stretches, each featuring homogeneous biological assemblages. Indirect gradient analysis revealed relationships between the pressure gradient and selected features of the macroinvertebrate and macrophyte community but not for diatoms or phytoplankton. 4. We identified biological metrics suitable for the quality classification of individual stretches or the entire river. Impoundment is the major hydromorphological alteration on the Danube but various metrics still responded significantly to differences in the morphological condition of sites not affected by impoundment. 5. A comparison of macroinvertebrate sampling techniques (airlift versus kick‐and‐sweep) revealed differences in how the acquired data reflect the effects of anthropogenic pressure. Biological metrics based only on kick‐and‐sweep sample data were insensitive to habitat deterioration in the heavily modified Upper Danube. 6. This study exemplifies the empirical approach of alternative benchmarking in intercalibration and offers practical solutions to some of the challenges of large river bioassessment.     

First steps in the Central-Baltic intercalibration exercise on lake macrophytes: where do we start?

The Water Framework Directive (WFD 2000) defines macrophytes as one of the biological groups required for the ecological assessment of European surface waters. Several indices for macrophyte assessment have been proposed or are currently in use by different European states. As a first step towards performing an intercalibration of these indices a common dataset was developed. This dataset contains abundance data on 789 macrophyte species from 316 different lake sites in ten European countries. Various common species and genera were identified as indicators of reference and impacted conditions within the dataset. Cluster analysis of macrophyte data, supported by non-metric multidimensional scaling, indicated that clusters formed were more reflective of their source country rather than lake type. This might be caused by differences in regional climate, biogeography, monitoring techniques, or a combination of these factors. A total of six national indices were applied to assign quality classes to the lakes. However, this produced results that often differed by one or two quality classes for the same site. We foresee that a more precise intercalibration exercise is necessary, and it should be based on more detailed data considering both seasonality and the latitudinal differences within the area covered.     

Comparing macroinvertebrate indices to detect organic pollution across Europe: a contribution to the EC Water Framework Directive intercalibration

With the EC Water Framework Directive (WFD) the ecological status of a water body is defined by comparing the observed biological community composition present with near-natural reference conditions. The ecological status is then classified into five quality classes (high, good, moderate, poor and bad). It is of great importance that `good ecological status' has the same meaning within the European Union, since water bodies not measuring up to these standards have to be improved. Therefore, the Ecological Quality Ratios (EQR) at high-good, and good-moderate quality class boundaries will be intercalibrated. Each country has to report physical, chemical, and biological data from two sites at each of these boundaries and since most data exist for benthic macroinvertebrates, this quality element will be of great importance in the intercalibration process. The aim of this study was therefore to compare the results of different benthic macroinvertebrate metrics used to assess the impact of organic pollution (including eutrophication) (one of the major human impacts on European streams). A selection of the data sampled in the AQEM project was evaluated, where benthic macroinvertebrate- and abiotic data from four countries (Austria, the Czech Republic, Portugal and Sweden) and seven `stream types' were included. An organic pollution (including eutrophication) gradient was defined using Principal Component Analysis and the boundaries for high-good and good-moderate ecological status set by the partners from each country were used to define arbitrary class boundaries. The Average Score Per Taxon (ASPT) was well correlated with the stress gradient in most stream types, whereas the Saprobic Index worked clearly better than ASPT in those countries (Austria and the Czech Republic) where macroinvertebrates are generally identified to lower (species) as opposed to a higher (genus or family) level of identification. Defining harmonised class boundaries is difficult; this process has to consider the natural differences between stream types (e.g. in the reference values of metrics) but has to eliminate different perceptions of ecological quality.     

A stressor specific multimetric approach for monitoring running waters in Austria using benthic macro-invertebrates

We investigated four stream types in four different bioregions, classified by catchment area and altitude, and stressed by different degrees of organic pollution and habitat alteration. We examined a macro-invertebrate based multimetric approach for Austrian rivers as a potential assessment method within the European Water Framework Directive. Benthic macro-invertebrate data (100 samples including reference sites) were used to develop a multimetric index for each stream type and targeted stressors. Sites were pre-classified based on physical, chemical, and land use criteria into five ecological quality classes. More than 200 biological metrics were tested for their sensitivity to the targeted stressors, their spatial and temporal variability and their ability to discriminate between different types and degrees of stress. Metrics for index development were selected to reflect different levels of information including ecosystem, community, and individual levels (Karr, 1991; Barbour et al., 1995; Gerritsen, 1995). Combinations of metrics were selected to distinguish best between non or slightly impaired and stressed sites (evaluated by calculating discrimination efficiency values and power analysis). The resulting four indices comprised seven to nine metrics from five to seven metric categories, and distinguished reference/slightly disturbed sites from stressed sites with close to 100% efficiency. The indices can form the basis for stressor-specific assessment of stream condition.     

Direct comparison of assessment methods using benthic macroinvertebrates: a contribution to the EU Water Framework Directive intercalibration exercise

The aim of the intercalibration exercise presently performed by the EU is to identify and resolve significant inconsistencies between the ecological quality classifications of EU Member States and the normative definitions of the EU Water Framework Directive. Based on benthic macroinvertebrate data of two European stream types (small siliceous mountain streams and medium-sized lowland streams in Central and Western Europe) we correlated the indices of 10 river quality assessment methods (ASPT, BMWP, DSFI, German Multimetric Index, Saprobic Indices) applied in Austria, Czech Republic, Denmark, Germany, Poland, Slovak Republic, Sweden and United Kingdom. National class boundaries were compared via regression analysis. Assessment methods of the same type (Saprobic Indices, BMWP/ASPT scores) showed best correlation results (R²>0.7). The good quality status boundaries of the national methods deviated up to 25%; thus indicating the necessity to harmonize the national classification schemes. Prerequisites of the presented intercalibration approach are (1) a sufficiently large and consistent dataset representative of the respective common intercalibration types and (2) agreement on common type specific reference conditions.     

The seagrass Posidonia oceanica as indicator of coastal water quality: Experimental intercalibration of classification systems

The pervasive use of ecological indices is increasingly requiring actions of harmonisation. Specifically, within the EU Water Framework Directive, an important effort in methods intercalibration is being done. However, a significant limitation in comparability assessment arises from the datasets used, which have different geographic origins. The purpose of our study was to perform an experimental intercalibration, where data were collected specifically on a set of common sites and following all the requirements of the methods being assessed. Three indices based on the marine angiosperm Posidonia oceanica, the POMI, the BiPo and the PoSte, were applied to sites in three different geographical areas of the western Mediterranean: Catalonia, Corsica and Southern Italy (Ischia), distant between hundreds and more than thousands of kilometers. Two indices, POMI and BiPo, showed not only a very good relationship with human pressures (measured on a common scale for all sites) but also a high comparability, in all aspects investigated. The differences found for the third one (PoSte) are hypothesised as being due to a different rationale used to define reference conditions, the different metrics used in the index, and in particular to a different definition of ecological status in relation to the time scale of the response to anthropogenic pressures. Our study demonstrates that indices with very different approaches can provide fully reliable and comparable results.     

The validation of common European class boundaries for river benthic macroinvertebrates to facilitate the intercalibration process of the Water Framework Directive

The Water Framework Directive (WFD) requires that all the water bodies in Europe be protected and enhanced to achieve Good Ecological Status by 2015. The intercalibration of the biological monitoring results of Member States has to be carried out in relation to classification tasks to guarantee a common understanding of ‘Good Ecological Status’ at a European level. An intercalibration exercise was carried out within the framework of different Geographical Intercalibration Groups (GIGs), each composed of a group of countries that share similar river types and have a discrete geographical range. By means of the intercalibration exercise, common European boundaries were proposed for river invertebrate assessment methods within each GIG. The purpose of this study was to validate the boundaries proposed for the formal intercalibration exercise. A benchmark data set was used, which comprised data, collected in different European countries that satisfied WFD requirements. The data set included a set of reference sites and provided evidence of a high degree of comparability among countries. The STAR Intercalibration Common Metric index (STAR_ICMi) was calculated for benchmark samples and was selected as the index in which national assessment boundaries were expressed. It was applied for the intercalibration exercise in two GIGs. For the STAR_ICMi, the coefficient of variation was also calculated, demonstrating a comparable variability with indices that are based on species level identification. A fixed percentile of reference samples STAR_ICMi values was selected as the boundary between High and Good status. The range from this fixed percentile to the lowest possible value was divided into four equal parts to obtain the remaining class boundaries. The resulting High/Good and Good/Moderate boundaries were compared to the boundaries proposed by GIGs and proved to be in line with those defined in the intercalibration exercise. Even if, for the intercalibration exercise, some procedures to check the Member States (MS) data sets where put into practice, it is the responsibility of each state to guarantee the WFD compliancy of their reference conditions and methods. Accordingly, the process of validation explained in this article, or similar ones, can be an important step forward and demonstrate the comparability of the actual boundaries.     

Comparing aspirations: intercalibration of ecological status concepts across European lakes for littoral diatoms

Eleven European countries participated in an exercise to harmonise diatom-based methods used for status assessment in lakes. Lakes were divided into low, medium and high alkalinity types for this exercise. However, it was not possible to perform a full intercalibration on low alkalinity lakes due to the short gradient and confounding factors. Values of the Trophie Index were computed for all samples in order that national datasets could all be expressed on a common scale. Not all participants had reference sites against which national methods could be standardised and, therefore, a Generalised Linear Modelling approach was used to control the effect of national differences in datasets. This enabled the high/good and good/moderate status boundaries to be expressed on a common scale and for deviations beyond ±0.25 class widths to be identified. Those countries which had relaxed boundaries were required to adjust these to within ±0.25 class widths whilst the intercalibration rules allowed those countries with more stringent boundaries to retain these. Despite biogeographical and typological differences between countries, there was broad agreement on the characteristics of high, good and moderate status diatom assemblages, and the exercise has ensured consistent application of Water Framework Directive assessments around Europe.     

A comparison of national approaches to setting ecological status boundaries in phytobenthos assessment for the European Water Framework Directive: results of an intercalibration exercise

The European Union (EU)’s Water Framework Directive (WFD) requires that all Member States participate in intercalibration exercises in order to ensure that ecological status concepts and assessment levels are consistent across the EU. This paper describes one such exercise, performed by the countries in the Central/Baltic Geographical Intercalibration Group stretching from Ireland in the west to Estonia in the east and from the southern parts of Scandinavia to the northern regions of Spain and Italy (but excluding alpine regions, which were intercalibrated separately). In this exercise, methods used to measure ecological status of rivers using benthic diatoms were compared. Ecological status is estimated as the ratio between the observed value of a biological element and the value expected in the absence of significant human impact. Approaches to defining the ‘reference sites’, from which these ‘expected’ values were derived, varied from country to country. Minimum criteria were established as part of the exercise but there was still considerable variation between national reference values, reflecting typological differences that could not be resolved during the exercise. A simple multimetric index was developed to compare boundary values using two widely used diatom metrics. Boundary values for high/good status and good/moderate status set by each participant were converted to their equivalent values of this intercalibration metric using linear regression. Variation of ±0.05 EQR units around the median value was considered to be acceptable and the exercise provided a means for those Member States who fell significantly above or below this line to review their approaches and, if necessary, adjust their boundaries. Handling editor: J. Padisak     

Assessment metrics for littoral seagrass under the European Water Framework Directive; outcomes of UK intercalibration with the Netherlands

Under the Water Framework Directive (WFD) taxonomic composition, presence of disturbance sensitive species, abundance and cover are stated attributes for monitoring the status of marine angiosperms; a biological quality element required for assessment of environmental condition. Member States (MS) are required to devise metrics for assessing these attributes to establish ecological status of water bodies. Furthermore the Directive requires intercalibration of metrics and data between MSs. Seagrass are the only truly marine angiosperms. The proposed suites of UK and Dutch metrics for assessing the specified seagrass attributes are described and comparisons made. UK and Dutch metrics are intercalibrated through testing against each nationality’s seagrass data. Strong agreement is established in the outcomes of the two suites of tools; >83%. Differences in outcomes are usually due to lack of availability of raw data. Importantly, where outcomes differ they still fall on the same side of the Good/Moderate boundary. Electronic Supplementary Material Supplementary material is available in the online version of this article at and is accessible for authorized users Handling editor: K. Martens An erratum to this article is available at .     

Using biological metrics to score and evaluate sites: a nearest-neighbour reference condition approach

1. Reference (i.e. least or minimally impaired) sites can provide important information about the expected range of biological metrics and can be used to establish impairment or non‐impairment of a test site. A problem with using reference data is that biological metrics are affected by natural conditions. We present an approach that uses local information to adjust for natural conditions and a method for statistically evaluating condition at a test site using biological metrics. 2. Our method consists of four steps: selection of a distance measure to find neighbours of a test site, selecting natural variables to measure the distance, selection of the number of neighbours and calculating a scored metric. 3. We use a simulated example to illustrate when the nearest‐neighbour approach improves classification of sites as reference or not reference. 4. Using a set of data from the Mid‐Atlantic Highlands, we show that the nearest‐neighbour method improved on the ability of a regression approach to correctly classify test sites known to be from a non‐reference group without affecting the ability to correctly classify test sites known to be from the reference group.     

Intercalibration of assessment methods for macrophytes in lowland streams: direct comparison and analysis of common metrics

The results of four macrophyte assessment methods (French Indice Biologique Macrophytique en Rivière, German Reference Index, British Mean Trophic Rank and Dutch Macrophyte Score) were compared, based on plant survey data of medium-sized lowland streams in Central Europe. To intercalibrate the good quality class boundaries two alternative methods were applied: direct comparison and the use of “common metrics”. While the French and British methods were highly related (R²>0.75), the German RI showed less (0.20<R²<0.55) and the Dutch DMS least correlation (R²<0.10) with other methods. Of 70 macrophyte metrics tested only Ellenberg_N was considerably related to three of the national assessment methods, thus representing a potential common metric for intercalibration. Comparison of quality class boundaries via regression analysis using both intercalibration approaches revealed major differences between classifications of the French, German and British methods, which are, in addition, related in a nonlinear way.     

Modeling wetland plant metrics to improve the performance of vegetation-based indices of biological integrity

The objective of this study was to determine if the accuracy and precision of wetland plant indices of biological integrity (IBIs) could be improved through the use of modeling techniques. To do this, we developed a modeled vegetation IBI (MVIBI) based on metrics previously used to develop vegetation indices of biological integrity (VIBIs) for Ohio wetlands (e.g. % invasive grass, % sensitive species, shrub richness). We selected 82 emergent, forested, and shrub-dominated reference sites distributed across the State of Ohio and built Random Forest models to predict plant metric scores at reference wetlands from naturally occurring environmental features related to climate, hydrology, geology, soils, and landscape position. The models explained between 14 and 52% of the variance in the scores of 21 metrics indicating that variation in wetland plant assemblages was significantly associated with naturally occurring environmental gradients. We used principal component analysis to identify ten groups of statistically independent metrics and selected one metric from each group that discriminated most strongly between reference and most degraded sites based on t-scores. Two axes did not contain discriminating metrics so we used eight metrics in the MVIBI. Analysis of variance of reference site MVIBI scores indicated that we could use one distribution of reference site scores to assess multiple wetland types, thus eliminating the need to separately designate wetland types. We used the MVIBI to assess 170 test sites and compared the accuracy, precision, responsiveness, and sensitivity of the MVIBI to those of the original VIBIs. The MVIBI was up to twice as accurate and precise as the original VIBIs, indicating that modeling can be used to improve the performance of vegetation-based IBIs. The use of model-based IBIs for wetland plants should reduce assessment errors associated with natural variation in plant metrics and should increase confidence in wetland assessments.     

Macroinvertebrate-based multimetric predictive models for evaluating the human impact on biotic condition of Bolivian streams

We developed and validated a single multimetric index based on predictive models that could evaluate anthropogenic disturbances in streams of three disparate ecoregions of Bolivia. To do so, we examined 45 candidate metrics reflecting different aspects of macroinvertebrate assemblage structure and function gleaned from available literature and for their potential to indicate degradation. More importantly, we integrated functional trait metrics to improve the sensitivity of our index. To quantify possible deviation from reference conditions, we first established and validated statistical models describing metric responses to natural environmental differences in the absence of any significant anthropogenic disturbance. We considered that the residual distributions of these models described the response range of each metric, independently of natural environmental influence. After testing the sensitivity of these residuals to a gradient of anthropogenic disturbance, we retained eight metrics that were used in the final assemblage index, four metrics based on richness and composition and four metrics based on biological traits. Our index performed well in discriminating between reference and disturbed sites, giving a significant negative linear response to a gradient of physical and chemical anthropogenic disturbances. After employing a probability survey design and sampling a relatively small number of sites throughout all major ecoregions of Bolivia, we believe our methodology can be used to develop a monitoring tool to evaluate status and trends in biological condition for streams of the entire country despite its complex and heterogeneous geology and climate.