The joint effects of selection and dominance on the QST - FST contrast

Q_ST measures the differentiation of quantitative traits between populations. It is often compared to F_ST, which measures population differentiation at neutral marker loci due to drift, migration, and mutation. When Q_ST is different from F_ST, it is usually taken as evidence that selection has either restrained or accelerated the differentiation of the quantitative trait relative to neutral markers. However, a number of other factors such as inbreeding, dominance, and epistasis may also affect the Q_ST − F_ST contrast. In this study, we examine the effects of dominance, selection, and inbreeding on Q_ST − F_ST. We compare Q_ST with F_ST at selected and neutral loci for populations at equilibrium between selection, drift, mutation, and migration using both analytic and simulation approaches. Interestingly, when divergent selection is acting on a locus, inbreeding and dominance generally inflate Q_ST relative to F_ST when they are both measured at the quantitative locus at equilibrium. As a consequence, dominance is unlikely to hide the signature of divergent selection on the Q_ST − F_ST contrast. However, although in theory dominance and inbreeding affect the expectation for Q_ST − F_ST, of most concern is the very large variance in both Q_ST and F_ST, suggesting that we should be cautious in attributing small differences between Q_ST and F_ST to selection.