Classification of early‐stage non‐small cell lung cancer by weighing gene expression profiles with connectivity information

Pathway‐based feature selection algorithms, which utilize biological information contained in pathways to guide which features/genes should be selected, have evolved quickly and become widespread in the field of bioinformatics. Based on how the pathway information is incorporated, we classify pathway‐based feature selection algorithms into three major categories—penalty, stepwise forward, and weighting. Compared to the first two categories, the weighting methods have been underutilized even though they are usually the simplest ones. In this article, we constructed three different genes’ connectivity information‐based weights for each gene and then conducted feature selection upon the resulting weighted gene expression profiles. Using both simulations and a real‐world application, we have demonstrated that when the data‐driven connectivity information constructed from the data of specific disease under study is considered, the resulting weighted gene expression profiles slightly outperform the original expression profiles. In summary, a big challenge faced by the weighting method is how to estimate pathway knowledge‐based weights more accurately and precisely. Only until the issue is conquered successfully will wide utilization of the weighting methods be impossible.     

Introducing weights to life cycle sustainability assessment—how do decision-makers weight sustainability dimensions?

Purpose

Decisions based on life cycle sustainability assessment (LCSA) pose a multi-criteria decision issue, as impacts on the three different sustainability dimensions have to be considered which themselves are often measured through several indicators. To support decision-making at companies, a method to interpret multi-criteria assessment and emerging trade-offs would be beneficial. This research aims at enabling decision-making within LCSA by introducing weights to the sustainability dimensions.

Methods

To derive weights, 54 decision-makers of different functions at a German automotive company were asked via limit conjoint analysis how they ranked the economic, environmental, and social performance of a vehicle component. Results were evaluated for the entire sample and by functional clusters. Additionally, sustainability respondents, i.e., respondents that dealt with sustainability in their daily business, were contrasted with non-sustainability respondents. As a last step, the impact of outliers was determined. From this analysis, practical implications for ensuring company-optimal decision-making in regard to product sustainability were derived.

Results and discussion

The results showed a large spread in weighting without clear clustering. On average, all sustainability dimensions were considered almost equally important: the economic dimension tallied at 33.5%, the environmental at 35.2%, and the social at 31.2%. Results were robust as adjusting for outliers changed weights on average by less than 10%. Results by function showed low consistency within clusters hinting that weighting was more of a personal than a functional issue. Sustainability respondents weighted the social before the environmental and economic dimension while non-sustainability respondents put the economic before the other two dimensions. Provided that the results of this research could be generalized, the retrieved weighting set was seen as a good way to introduce weights into an operationalized LCSA framework as it represented the quantification of the already existing decision process. Therefore, the acceptance of this weighting set within the respective company was expected to be increased.

Conclusions

It could be shown that conjoint analysis enabled decision-making within LCSA by introducing weights to solve a multi-criteria decision issue. Furthermore, implications for practitioners could be derived to ensure company-optimal decision-making related to product sustainability. Future research should look at expanding the sample size and geographical scope as well as investigating the weighting of indicators within sustainability dimensions and the drivers that influence personal decision-making in regard to weighting sustainability dimensions.

     

Discovering Associations in Biomedical Datasets by Link-based Associative Classifier (LAC)

Associative classification mining (ACM) can be used to provide predictive models with high accuracy as well as interpretability. However, traditional ACM ignores the difference of significances among the features used for mining. Although weighted associative classification mining (WACM) addresses this issue by assigning different weights to features, most implementations can only be utilized when pre-assigned weights are available. In this paper, we propose a link-based approach to automatically derive weight information from a dataset using link-based models which treat the dataset as a bipartite model. By combining this link-based feature weighting method with a traditional ACM method–classification based on associations (CBA), a Link-based Associative Classifier (LAC) is developed. We then demonstrate the application of LAC to biomedical datasets for association discovery between chemical compounds and bioactivities or diseases. The results indicate that the novel link-based weighting method is comparable to support vector machine (SVM) and RELIEF method, and is capable of capturing significant features. Additionally, LAC is shown to produce models with high accuracies and discover interesting associations which may otherwise remain unrevealed by traditional ACM.     

On the bias produced by quality scores in meta-analysis,and a hierarchical view of proposed solutions

Results from better quality studies should in some sense be more valid or more accurate than results from other studies, and as a consequence should tend to be distributed differently from results of other studies. To date, however, quality scores have been poor predictors of study results. We discuss possible reasons and remedies for this problem. It appears that 'quality' (whatever leads to more valid results) is of fairly high dimension and possibly non-additive and nonlinear, and that quality dimensions are highly application-specific and hard to measure from published information. Unfortunately, quality scores are often used to contrast, model, or modify meta-analysis results without regard to the aforementioned problems, as when used to directly modify weights or contributions of individual studies in an ad hoc manner. Even if quality would be captured in one dimension, use of quality scores in summarization weights would produce biased estimates of effect. Only if this bias were more than offset by variance reduction would such use be justified. From this perspective, quality weighting should be evaluated against formal bias-variance trade-off methods such as hierarchical (random-coefficient) meta-regression. Because it is unlikely that a low-dimensional appraisal will ever be adequate (especially over different applications), we argue that response-surface estimation based on quality items is preferable to quality weighting. Quality scores may be useful in the second stage of a hierarchical response-surface model, but only if the scores are reconstructed to maximize their correlation with bias.     

On the use of weighting in LCA: translating decision makers’ preferences into weights via linear programming

Purpose

The main goal of any life cycle assessment (LCA) study is to identify solutions leading to environmental savings. In conventional LCA studies, practitioners select from some alternatives the one which better matches their preferences. This task is sometimes simplified by ranking these alternatives using an aggregated indicator defined by attaching weights to impacts. We address here the inverse problem. That is, given an alternative, we aim to determine the weights for which that solution becomes optimal.

Methods

We propose a method based on linear programming (LP) that determines, for a given alternative, the ranges within which the weights attached to a set of impact metrics must lie so that when a weighting combination of these impacts is optimized, the alternative can be optimal, while if the weights fall outside this range, it is guaranteed that the solution will be suboptimal. A large weight value implies that the corresponding LCA impact is given more importance, while a low value implies the converse. Furthermore, we provide a rigorous mathematical analysis on the implications of using weighting schemes in LCA, showing that this practice guides decision-making towards the adoption of some specific alternatives (those lying on the convex envelope of the resulting trade-off curve).

Results and discussion

A case study based on the design of hydrogen infrastructures is taken as a test bed to illustrate the capabilities of the approach presented. Given are a set of production and storage technologies available to produce and deliver hydrogen, a final demand, and cost and environmental data. A set of designs, each achieving a unique combination of cost and LCA impact, is considered. For each of them, we calculate the minimum and maximum weight to be given to every LCA impact so that the alternative can be optimal among all the candidate designs. Numerical results show that solutions with lower impact are selected when decision makers are willing to pay larger monetary penalties for the environmental damage caused.

Conclusions

LP can be used in LCA to translate the decision makers’ preferences into weights. This information is rather valuable, particularly when these weights represent economic penalties, as it allows screening and ranking alternatives on the basis of a common economic basis. Our framework is aimed at facilitating decision making in LCA studies and defines a general framework for comparing alternatives that show different performance in a wide variety of impact metrics.     

Two-dimensional grey cloud clustering-fuzzy entropy comprehensive assessment model for river health evaluation

Abstract

The river health evaluation is typically complex non-linear system with characteristics of fuzziness and randomness. However, conventional gray clustering method has difficult to effectively describe fuzzy and random information simultaneously. For this purpose, the cloud model and fuzzy entropy theory are introduced to establish 2D gray cloud clustering-fuzzy entropy comprehensive evaluation model. Different with health level models, it reflects river health situation from aspects of health level and corresponding water body complexity simultaneously. The health level is obtained by gray cloud whitened weight function (first sub-system) and fuzzy entropy represents complexity and fuzziness of river health situation (second sub-system). Moreover, multi-level river health evaluation indicator system is constructed with dividing indicators into common and distinct sections according to differences on river characteristics. Meanwhile, indicator weights are determined by renewed combined weighting method based on minimum deviation principle. Finally, we conduct health evaluation work for rivers in the Taihu basin. The evaluation health levels and fuzzy entropy for river A–G are H3 (0.4888, relatively significant); H2 (0.5476, relatively fuzzy); H2 (0.7526, fuzzy); H2 (0.4731, relatively significant); H2 (05138, relatively fuzzy); H3 (0.5822, relatively fuzzy), and H2 (0.4064, relatively significant), respectively. Results are consistent with current river health situation and more intuitive than compared models. Furthermore, evaluation results with four different weighting methods are compared to further demonstrate rationality of the weighting method and evaluation model. Hence, the model proposed is demonstrated to provide new insight for solving river health assessment problem effectively.     

Optimal weight in estimating and comparing areas under the receiver operating characteristic curve using longitudinal data

In the setting of longitudinal study, subjects are followed for the occurrence of some dichotomous outcome. In many of these studies, some markers are also obtained repeatedly during the study period. Emir et al. introduced a non-parametric approach to the estimation of the area under the ROC curve of a repeated marker. Their non-parametric estimate involves assigning a weight to each subject. There are two weighting schemes suggested in their paper: one for the case when within-patient correlation is low, and the other for the case when within-subject correlation is high. However, it is not clear how to assign weights to marker measurements when within-patient correlation is modest. In this paper, we consider the optimal weights that minimize the variance of the estimate of the area under the ROC curve (AUC) of a repeated marker, as well as the optimal weights that minimize the variance of the AUC difference between two repeated markers. Our results in this paper show that the optimal weights depend not only on the within-patient control--case correlation in the longitudinal data, but also on the proportion of subjects that become cases. More importantly, we show that the loss of efficiency by using the two weighting schemes suggested by Emir et al. instead of our optimal weights can be severe when there is a large within-subject control--case correlation and the proportion of subjects that become cases is small, which is often the case in longitudinal study settings.     

Support Weighting

Previous weighting methods—including compatibility weighting—have assumed that homoplasy indicates unreliability, but this assumption does not seem to hold for large molecular data matrices. Reliability can be better assessed by support weighting, which measures the degree to which the changes in a character (site) are concentrated in the supported branches of a tree. Jackknife resampling can be used to generate randomly selected suites of initial weights in successive support weighting, and this provides a way of assessing the stability of successive weighting results.     

A method for determining weights for excess relative risk and excess absolute risk when applied in the calculation of lifetime risk of cancer from radiation exposure

Radiation-related risks of cancer can be transported from one population to another population at risk, for the purpose of calculating lifetime risks from radiation exposure. Transfer via excess relative risks (ERR) or excess absolute risks (EAR) or a mixture of both (i.e., from the life span study (LSS) of Japanese atomic bomb survivors) has been done in the past based on qualitative weighting. Consequently, the values of the weights applied and the method of application of the weights (i.e., as additive or geometric weighted means) have varied both between reports produced at different times by the same regulatory body and also between reports produced at similar times by different regulatory bodies. Since the gender and age patterns are often markedly different between EAR and ERR models, it is useful to have an evidence-based method for determining the relative goodness of fit of such models to the data. This paper identifies a method, using Akaike model weights, which could aid expert judgment and be applied to help to achieve consistency of approach and quantitative evidence-based results in future health risk assessments. The results of applying this method to recent LSS cancer incidence models are that the relative EAR weighting by cancer solid cancer site, on a scale of 0–1, is zero for breast and colon, 0.02 for all solid, 0.03 for lung, 0.08 for liver, 0.15 for thyroid, 0.18 for bladder and 0.93 for stomach. The EAR weighting for female breast cancer increases from 0 to 0.3, if a generally observed change in the trend between female age-specific breast cancer incidence rates and attained age, associated with menopause, is accounted for in the EAR model. Application of this method to preferred models from a study of multi-model inference from many models fitted to the LSS leukemia mortality data, results in an EAR weighting of 0. From these results it can be seen that lifetime risk transfer is most highly weighted by EAR only for stomach cancer. However, the generalization and interpretation of radiation effect estimates based on the LSS cancer data, when projected to other populations, are particularly uncertain if considerable differences exist between site-specific baseline rates in the LSS and the other populations of interest. Definitive conclusions, regarding the appropriate method for transporting cancer risks, are limited by a lack of knowledge in several areas including unknown factors and uncertainties in biological mechanisms and genetic and environmental risk factors for carcinogenesis; uncertainties in radiation dosimetry; and insufficient statistical power and/or incomplete follow-up in data from radio-epidemiological studies.     

Extended implied weighting

Several extensions to implied weighting, recently implemented in TNT, allow a better treatment of data sets combining morphological and molecular data sets, as well as those comprising large numbers of missing entries (e.g. palaeontological matrices, or combined matrices with some genes sequenced for few taxa). As there have been recent suggestions that molecular matrices may be better analysed using equal weights (rather than implied weighting), a simple way to apply implied weighting to only some characters (e.g. morphology), leaving other characters with a constant weight (e.g. molecules), is proposed. The new methods also allow weighting entire partitions according to their average homoplasy, giving each of the characters in the partition the same weight (this can be used for dynamically weighting, e.g. entire genes, or first, second, and third positions collectively). Such an approach is easily implemented in schemes like successive weighting, but in the case of implied weighting poses some particular problems. The approach has the peculiar implication that the inclusion of uninformative characters influences the results (by influencing the implied weights for the partitions). Last, the concern that characters with many missing entries may receive artificially inflated weights (because they necessarily display less homoplasy) can be solved by allowing the use of different weighting functions for different characters, in such a way that the cost of additional transformations decreases more rapidly for characters with more missing entries (thus effectively assuming that the unobserved entries are likely to also display some unobserved homoplasy). The conceptual and practical aspects of all these problems, as well as details of the implementation in TNT, are discussed.     

Improving power of genome-wide association studies with weighted false discovery rate control and prioritized subset analysis

The issue of large-scale testing has caught much attention with the advent of high-throughput technologies. In genomic studies, researchers are often confronted with a large number of tests. To make simultaneous inference for the many tests, the false discovery rate (FDR) control provides a practical balance between the number of true positives and the number of false positives. However, when few hypotheses are truly non-null, controlling the FDR may not provide additional advantages over controlling the family-wise error rate (e.g., the Bonferroni correction). To facilitate discoveries from a study, weighting tests according to prior information is a promising strategy. A 'weighted FDR control' (WEI) and a 'prioritized subset analysis' (PSA) have caught much attention. In this work, we compare the two weighting schemes with systematic simulation studies and demonstrate their use with a genome-wide association study (GWAS) on type 1 diabetes provided by the Wellcome Trust Case Control Consortium. The PSA and the WEI both can increase power when the prior is informative. With accurate and precise prioritization, the PSA can especially create substantial power improvements over the commonly-used whole-genome single-step FDR adjustment (i.e., the traditional un-weighted FDR control). When the prior is uninformative (true disease susceptibility regions are not prioritized), the power loss of the PSA and the WEI is almost negligible. However, a caution is that the overall FDR of the PSA can be slightly inflated if the prioritization is not accurate and precise. Our study highlights the merits of using information from mounting genetic studies, and provides insights to choose an appropriate weighting scheme to FDR control on GWAS.     

An analytic framework for social life cycle impact assessment—part 1: methodology

Purpose

This study aims to develop a new framework of social life cycle impact assessment (SLCIA) method based on the United Nations Environment Program/Society of Environmental Toxicology and Chemistry (UNEP/SETAC) Guidelines for analyzing the social impact in Taiwan, particularly in the electronics industry.

Methods

After reviewing the literature on social life cycle assessment (SLCA), we analyzed existing case studies and developed SLCIA methods based on the UNEP/SETAC Guidelines. We thereafter identified stakeholders, subcategories, and indicators in accordance with the current status of SLCA case studies and opinions from ten experts in the Taiwanese electronics industry. Both quantitative and semi-quantitative indicators were subsequently proposed to assess the social impact of workers in the Taiwanese electronics sector. Each indicator was given the score of 1 to 5 by classifying the social impact percentage of nine scales. To formulate an analytic framework for SLCIA, the weighting values of each subcategory and indicator were determined using the consistent fuzzy preference relations (CFPR) method.

Results and discussion

Seven subcategories and 19 qualitative and quantitative indicators of worker stakeholders for the electronics sector were identified based on the UNEP/SETAC Guidelines. A score of 1 to 5 is assigned to each quantitative indicator by classifying the social impact percentage of nine scales. The data obtained from companies for each quantitative indicator were subsequently transformed into social impact percentage in terms of the statistical data on social situations at the country or industry level. With regard to semi-quantitative indicators, three implementation levels of management efforts on social performance within five elements were identified. The CFPR method was then employed to determine the weights of each indicator by ten experts. Results indicated that preventing forced work practices, protecting children from having to work, and providing minimum and fair wages for workers are the three most important indicators for assessing social impact.

Conclusions

A new SLCIA method that incorporates both quantitative and semi-quantitative indicators was proposed for assessing social impact in the electronics sector in Taiwan. Nine quantitative indicators can be easily organized using available social data from government statistics as performance reference points (PRPs) to determine the social impact exerted by companies. The relative weights were determined to allow for an impact assessment and thus solve the limitation of their currently assumed equal weights. The proposed framework is examined to analyze the social impact of three production sites for semiconductor packaging and manufacturing in Taiwan.

     

Review of criteria for evaluating LCA weighting methods

Purpose

In the process of selecting where effective environmental measures should be directed, the weighting step of life cycle assessment (LCA) is an optional, controversial, but nevertheless important tool. A set of criteria for evaluating weighting methods has relevance in the process of acquiring meta-knowledge, and thus competence, in assigning relative weights to environmental impact categories. This competence is helpful when choosing between presently available weighting methods, and in creating new weighting methods.

Methods

Criteria in LCA-related literature are reviewed. The authors have focused on identifying lists of criteria rather than extracting criteria from bulks of text. An important starting point has been the actual use of the term “criterion”, while at the same time disqualifying certain definitions of the term which are too far removed from the two definitions provided in this article.

Results and discussion

Criteria for evaluating weighting methods are shown to fall into two general categories. The first being general criteria for weighting methods, demanding that weighting methods have a broad scope, are practical for users and scientists, are scientific and have ethical goals. The second being criteria proposing characteristics of concrete environmental damage which should be taken into account by a weighting method. A noteworthy example is reversibility.

Conclusions

While the comprehensive tables of criteria speak for themselves, it can be observed that the need for transparency is particularly highlighted in literature. Furthermore, ISO 14044’s statement that the weighting step is “not scientifically based” would appear to defy a significant proportion of the other criteria reviewed; this, however, depends on its interpretation.     

Optimally weighted Z-test is a powerful method for combining probabilities in meta-analysis

The inverse normal and Fisher's methods are two common approaches for combining P-values. Whitlock demonstrated that a weighted version of the inverse normal method, or 'weighted Z-test', is superior to Fisher's method for combining P-values for one-sided T-tests. The problem with Fisher's method is that it does not take advantage of weighting and loses power to the weighted Z-test when studies are differently sized. This issue was recently revisited by Chen, who observed that Lancaster's variation of Fisher's method had higher power than the weighted Z-test. Nevertheless, the weighted Z-test has comparable power to Lancaster's method when its weights are set to square roots of sample sizes. Power can be further improved when additional information is available. Although there is no single approach that is the best in every situation, the weighted Z-test enjoys certain properties that make it an appealing choice as a combination method for meta-analysis.     

Evaluation of life cycle impacts: Identification of societal weights of environmental issues

A new method for identification of weights of environmental issues is suggested using the societal approach in the context of a weighting step in Life Cycle Assessment (LCA). The weights assigned by different economic groups to eleven environmental issues is obtained through analysis of linguistically stated relative rankings using a fuzzy partial ordering method. The system identification technique based on neural networks is used to identify logical connective in the stated relative rankings and this obviated the inconsistency problem normally encountered in the analysis of relative preference statements. The transitive property of a matrix of relative weights is used to minimise the number of responses to be elicited from a respondent.     

Character diversity and taxonomic weighting in the numerical taxonomy of two insect groups (Hemiptera: Diaspididae, and Lepidoptera: Drepanidae)

Abstract Methods used to assess ecological diversity have been adapted to show that binary and multistate taxonomic characters may be assigned measures of diversity that correspond closely with their information content. A distinction is drawn between the two major components of character diversity, corresponding respectively to the number of states exhibited and the evenness with which these are distributed over taxa. By choosing characters with high heterogeneity (i.e. high values of those indices which combine the two components of diversity) it is possible to select relatively small subsets of characters which effectively reproduce the main features of reference ordinations of the taxa which were based on all available characters. Such diversity indices and the appropriate measures of information content can be regarded as a system of weights which vary ten-fold or more among themselves and which can be applied to all the characters to produce numerically weighted ordinations. Despite the considerable differential weighting introduced in this way, the resulting ordiantions differ very little from 'unweighted' reference ordinations in which all characters have been accorded equal weights. The results suggest that numerical weighting of this kind does little to improve the quality of phenetic procedures when applied to reasonably large bodies of real taxonomic data.     

Phylogeny of the parasitic wasp subfamily Charipinae (Hymenoptera, Cynipoidea, Figitidae)

The Charipinae are a major group of hyperparasitoids of Hemiptera. Here, we present the first cladistic analysis of this subfamily's internal relationships, based on 96 morphological characters of adults. The data matrix was analysed using uniformly weighted parsimony. The effects of using alternative weighting schemes were explored by performing additional searches employing implied weights criteria. One of the caveats of implied weights analysis is that it lacks an objective criterion for selecting the value of the concavity function. In the present study, differential weighting was used to explore the sensitivity of our results to the alternative assumptions made in the analysis and to select one of the most parsimonious trees under equal weights, which we regard as being the hypothesis that minimizes the amount of ad hoc assumptions. The validity of the two existing tribes and the monophyly of all the genera of Charipinae were tested, in particular the cosmopolitan and highly species-rich Alloxysta and Phaenoglyphis , which appear repeatedly in ecological and biochemical studies of host–parasitoid associations. The evolution of several major characters and the relationships between genera are discussed. On the basis of the phylogenetic results, we discuss a number of taxonomic issues. A new classification of the subfamily is proposed in which no tribes are maintained, Carvercharips is synonymyzed with Alloxysta , and the creation of a new genus from Nepal is justified. Our analysis points to the need for a world revision of the basal genus Phaenoglyphis , which is shown as paraphyletic.     

Implied weighting and its utility in palaeontological datasets: a study using modelled phylogenetic matrices

Implied weighting, a method for phylogenetic inference that actively seeks to downweight supposed homoplasy, has in recent years begun to be widely utilized in palaeontological datasets. Given the method's purported ability at handling widespread homoplasy/convergence, we investigate the effects of implied weighting on modelled phylogenetic data. We generated 100 character matrices consisting of 55 characters each using a Markov Chain morphology model of evolution based on a known phylogenetic tree. Rates of character evolution in these datasets were variable and generated by pulling from a gamma distribution for each character in the matrix. These matrices were then analysed under equal weighting and four settings of implied weights (k = 1, 3, 5, and 10). Our results show that implied weighting is inconsistent in its ability to retrieve a known phylogenetic tree. Equally weighted analyses are found to generally be more conservative, retrieving higher frequency of polytomies but being less likely to generate erroneous topologies. Implied weighting is found to generally resolve polytomies while also propagating errors, resulting in an increase in both correctly and incorrectly resolved nodes with a tendency towards higher rates of error compared to equal weighting. Our results suggest that equal weights may be a preferable method for parsimony analysis.     

Undesirable specialization in the construction of composite policy indicators: The Environmental Performance Index

The non-parametric Data Envelopment Analysis approach is increasingly used to construct composite indicators for country performance monitoring, benchmarking, and policy evaluation in a large variety of fields. The flexibility in the definition of aggregation weights is praised as the method's most important advantage: DEA allows each evaluated country to look for its own optimal weights that maximize the composite indicator relative to the other countries. However, this flexibility also carries a potential disadvantage as it may allow countries to appear as a brilliant performer in a manner that is hard to justify: by ignoring or overemphasizing one or multiple of the judiciously selected performance indicators. To illustrate this issue of undesirable specialization in DEA-based evaluations, this paper compares the Environmental Performance Index (EPI) as computed by the optimistic and pessimistic version of the DEA-model as proposed by Zhou et al. (2007). Based on both computed composites, undesirable specialization in performance is identified.     

How to tell green from grey? Towards a methodological framework for evaluating the greening of national tax systems

In this paper, we evaluate four types of indicators that can be used for measuring the greening of a tax system: revenue-based indicators, single tax rates, aggregate tax-rate based indicators and the implicit tax rate on energy. We develop an evaluation framework, introducing two principal evaluation criteria: content validity and comprehensiveness, and four statistical criteria: data availability, comparison over time, international comparability and ease of aggregation. Additional analysis regarding the issue of weighting is carried out for the aggregate tax-rate based indicator. The theoretical and methodological evaluation is supplemented and validated empirically using recent data on the Belgian and Flemish tax system. Finally, conclusions are drawn with regard to the strengths and the weaknesses of the four types of indicators, and recommendations are made for further research.