Neighbour joining trees, dominant markers and population genetic structure

Population genetic theory for 'traditional' codominant loci showing low levels of allelic diversity (eg allozymes) has been well characterised and evaluated. In contrast, appropriate methods for the analysis of data from more recently developed marker systems are still being refined. For multilocus dominant markers such as amplified fragment length polymorphisms (AFLPs) and randomly amplified polymorphic DNA (RAPDs), the methods of data analysis can be split into two main categories. In population-based approaches, population allele frequencies are compared to obtain some measure of the partitioning of genetic diversity into within- and between-population components. In contrast, individual-based approaches use individual multilocus genotypes as the unit of analysis. Inferences on population processes such as gene flow are based on inter-relationships among individual samples as visualised on phenetic diagrams such as neighbour joining trees. Using a simulation approach coupled with neighbour joining analyses, we show that while the underlying population genetic structure is an important determinant of tree shape in the analysis of dominant data, the number of loci examined also affects the topology. At low levels of population differentiation (eg FST=0.07), mutually exclusive clustering of individuals into their respective populations can occur when sufficiently large numbers of loci are scored (eg 250 loci, typical of many AFLP studies). In contrast, unresolved star-shaped topologies can be recovered at higher levels of population differentiation (FST= >0.15) when lower numbers of loci are employed (eg 50 loci, typical of many RAPD studies). Thus, the relationship between tree topology and the extent of genetic structuring of populations is contingent upon the number of dominant loci scored. The consequences of these findings for the biological interpretation of individual-based analysis of dominant data sets are discussed.