Identifying outlier loci in admixed and in continuous populations using ancestral population differentiation statistics

Finding genetic signatures of local adaptation is of great interest for many population genetic studies. Common approaches to sorting selective loci from their genomic background focus on the extreme values of the fixation index, F_ST, across loci. However, the computation of the fixation index becomes challenging when the population is genetically continuous, when predefining subpopulations is a difficult task, and in the presence of admixed individuals in the sample. In this study, we present a new method to identify loci under selection based on an extension of the F_ST statistic to samples with admixed individuals. In our approach, F_ST values are computed from the ancestry coefficients obtained with ancestry estimation programs. More specifically, we used factor models to estimate F_ST, and we compared our neutrality tests with those derived from a principal component analysis approach. The performances of the tests were illustrated using simulated data and by re‐analysing genomic data from European lines of the plant species Arabidopsis thaliana and human genomic data from the population reference sample, POPRES.