Identification of Ancestral Genes That Introduce Incongruence between Protein- and Species Trees

Two new approaches to detecting potential incongruence between a protein family tree and a species tree are considered. The first approach is based on the substitution of a known mapping of the gene tree G into the species tree S with a somewhat analogous multivalued G-into-S mapping. The second approach is based on the elementary concepts of the fuzzy set theory. Two algorithms corresponding to these approaches are described in detail, and their implementation is shown using a simulation example and three protein families from the database of clusters of orthologous protein groups (COGs).