The effect of gene flow on the coalescent time in the human-chimpanzee ancestral population

The coalescent process in the human-chimpanzee ancestral population is investigated using a model, which incorporates a certain time period of gene flow during the speciation process. a is a parameter to represent the degree and time of gene flow, and the model is identical to the null model with an instantaneous species split when a=infinity. A maximum likelihood (ML) method is developed to estimate a, and its power and reliability is investigated by coalescent simulations. The ML method is applied to nucleotide divergence data between human and chimpanzee. It is found that the null model with an instantaneous species split explains the data best, and no strong evidence for gene flow is detected. The result is discussed in the view of the mode of speciation. Another ML method is developed to estimate the male-female ratio (alpha) of mutation rate, in which the coalescent process in the ancestral population is taken into account.