Evolutionarily stable mixed mating in a variety of genetic systems

Many species display a mixture of close inbreeding and outbreeding which is referred to as mixed mating. For selfing species, models predict that such mixed mating systems can be stable. Conversely, models considering separate sex species have not been able to explain mixed mating systems. This failure may be a result of the unrealistic assumption that recurrent inbreeding does not increase the inbreeding coefficient. Here we show that mixed mating is expected in separate sex systems when recurrent inbreeding is taken into account. A female that allows her brother to sibmate with her gives an extra mating opportunity to said brother. This kin selective advantage should be strongest in genetic systems where the male is more related to the female. In support of this idea, we find that inbreeding evolves most easily in selfers, followed by diploid sibmating, followed by haplodiploid sibmating. Consideration of published values for the regression of fitness on inbreeding coefficient suggests that many species fall in a range where some selfing/sibmating is optimal.