Antagonistic evolution in an aposematic predator–prey signaling system

Warning signals within species, such as the bright colors of chemically defended animals, are usually considered mutualistic, monomorphic traits. Such a view is however increasingly at odds with the growing empirical literature, showing nontrivial levels of signal variation within prey populations. Key to understanding this variation, we argue, could be a recognition that toxicity levels frequently vary within populations because of environmental heterogeneity. Inequalities in defense may undermine mutualistic monomorphic signaling, causing evolutionary antagonism between loci that determine appearance of less well‐defended and better defended prey forms within species. In this article, we apply a stochastic model of evolved phenotypic plasticity to the evolution of prey signals. We show that when toxicity levels vary, then antagonistic interactions can lead to evolutionary conflict between alleles at different signaling loci, causing signal evolution, “red queen‐like” evolutionary chase, and one or more forms of signaling equilibria. A key prediction is that variation in the way that predators use information about toxicity levels in their attack behaviors profoundly affects the evolutionary characteristics of the prey signaling systems. Environmental variation is known to cause variation in many qualities that organisms signal; our approach may therefore have application to other signaling systems.