A likelihood-based approach to estimating and testing for isolation by distance

Simple regression of genetic similarities between pairs of populations on their corresponding geographic distances is frequently used to detect the presence of isolation by distance (IBD). However, these pairwise values are obviously not independent and there is no parametric procedure for estimating and testing for the IBD intercepts and slopes based on standard regression theory. Nonparametric tests, such as the Mantel test, and resampling techniques, such as bootstrapping, have been exploited with limited success. Here, I describe a likelihood-based analysis to allow for simultaneously detecting patterns of correlated residuals and estimating and testing for the presence of IBD. It is shown, through the analysis of two molecular datasets in pine species, that different covariance structures of the residuals exist. More over, the likelihood ratio tests under these covariance structures are less sensitive to the presence of IBD than the Mantel test and the simple regression analysis but more sensitive than the bootstrap and jackknife samples over independent populations or population pairs. Because the likelihood analysis directly models and accounts for nonindependence of residuals, it should legitimately detect the presence of IBD, thereby allowing for accurate inferences about evolutionary and demographic processes influencing the extent and patterns of IBD.