A heterogeneity test for fine-scale genetic structure

For organisms with limited vagility and/or occupying patchy habitats, we often encounter nonrandom patterns of genetic affinity over relatively small spatial scales, labelled fine-scale genetic structure. Both the extent and decay rate of that pattern can be expected to depend on numerous interesting demographic, ecological, historical, and mating system factors, and it would be useful to be able to compare different situations. There is, however, no heterogeneity test currently available for fine-scale genetic structure that would provide us with any guidance on whether the differences we encounter are statistically credible. Here, we develop a general nonparametric heterogeneity test, elaborating on standard autocorrelation methods for pairs of individuals. We first develop a 'pooled within-population' correlogram, where the distance classes (lags) can be defined as functions of distance. Using that pooled correlogram as our null-hypothesis reference frame, we then develop a heterogeneity test of the autocorrelations among different populations, lag-by-lag. From these single-lag tests, we construct an analogous test of heterogeneity for multilag correlograms. We illustrate with a pair of biological examples, one involving the Australian bush rat, the other involving toadshade trillium. The Australian bush rat has limited vagility, and sometimes occupies patchy habitat. We show that the autocorrelation pattern diverges somewhat between continuous and patchy habitat types. For toadshade trillium, clonal replication in Piedmont populations substantially increases autocorrelation for short lags, but clonal replication is less pronounced in mountain populations. Removal of clonal replicates reduces the autocorrelation for short lags and reverses the sign of the difference between mountain and Piedmont correlograms.