Severe outbreeding and inbreeding depression maintain mating system differentiation in Epipactis (Orchidaceae)

In hermaphroditic plants, theory for mating system evolution predicts that populations will evolve to either complete autonomous selfing (AS) or complete outcrossing, depending on the balance between automatic selection favouring self‐fertilization and costs resulting from inbreeding depression (ID). Theory also predicts that selection for selfing can occur rapidly and is driven by purging of genetic load and the loss of ID. Therefore, selfing species are predicted to have low levels of ID or even to suffer from outbreeding depression (OD), whereas predominantly outcrossing species are expected to have high levels of ID. To test these predictions, we related the capacity of AS to the magnitude of early‐acting inbreeding or OD in both allogamous and autogamous species of the orchid genus Epipactis. For each species, the level of AS was assessed under controlled greenhouse conditions, whereas hand‐pollinations were performed to quantify early costs of inbreeding or OD acting at the level of fruit and seed production. In the autogamous species, the capacity of AS was high (> 0.72), whereas in the allogamous species AS was virtually absent (< 0.10). Consistent with our hypothesis, allogamous Epipactis species had significantly higher total ID (average: 0.46) than autogamous species, which showed severe costs of OD (average: ?0.45). Overall, our findings indicate that strong early‐acting ID represents an important mechanism that contributes to allogamy in Epipactis, whereas OD may maintain selfing in species that have evolved to complete selfing.