Evolution of learned strategy choice in a frequency-dependent game

In frequency-dependent games, strategy choice may be innate or learned. While experimental evidence in the producer-scrounger game suggests that learned strategy choice may be common, a recent theoretical analysis demonstrated that learning by only some individuals prevents learning from evolving in others. Here, however, we model learning explicitly, and demonstrate that learning can easily evolve in the whole population. We used an agent-based evolutionary simulation of the producer-scrounger game to test the success of two general learning rules for strategy choice. We found that learning was eventually acquired by all individuals under a sufficient degree of environmental fluctuation, and when players were phenotypically asymmetric. In the absence of sufficient environmental change or phenotypic asymmetries, the correct target for learning seems to be confounded by game dynamics, and innate strategy choice is likely to be fixed in the population. The results demonstrate that under biologically plausible conditions, learning can easily evolve in the whole population and that phenotypic asymmetry is important for the evolution of learned strategy choice, especially in a stable or mildly changing environment.     

Stochastic payoff evaluation increases the temperature of selection

We study stochastic evolutionary game dynamics in populations of finite size. Moreover, each individual has a randomly distributed number of interactions with other individuals. Therefore, the payoff of two individuals using the same strategy can be different. The resulting "payoff stochasticity" reduces the intensity of selection and therefore increases the temperature of selection. A simple mean-field approximation is derived that captures the average effect of the payoff stochasticity. Correction terms to the mean-field theory are computed and discussed.     

Effects of Conformism on the Cultural Evolution of Social Behaviour

Models of cultural evolution study how the distribution of cultural traits changes over time. The dynamics of cultural evolution strongly depends on the way these traits are transmitted between individuals by social learning. Two prominent forms of social learning are payoff-based learning (imitating others that have higher payoffs) and conformist learning (imitating locally common behaviours). How payoff-based and conformist learning affect the cultural evolution of cooperation is currently a matter of lively debate, but few studies systematically analyse the interplay of these forms of social learning. Here we perform such a study by investigating how the interaction of payoff-based and conformist learning affects the outcome of cultural evolution in three social contexts. First, we develop a simple argument that provides insights into how the outcome of cultural evolution will change when more and more conformist learning is added to payoff-based learning. In a social dilemma (e.g. a Prisoner’s Dilemma), conformism can turn cooperation into a stable equilibrium; in an evasion game (e.g. a Hawk-Dove game or a Snowdrift game) conformism tends to destabilize the polymorphic equilibrium; and in a coordination game (e.g. a Stag Hunt game), conformism changes the basin of attraction of the two equilibria. Second, we analyse a stochastic event-based model, revealing that conformism increases the speed of cultural evolution towards pure equilibria. Individual-based simulations as well as the analysis of the diffusion approximation of the stochastic model by and large confirm our findings. Third, we investigate the effect of an increasing degree of conformism on cultural group selection in a group-structured population. We conclude that, in contrast to statements in the literature, conformism hinders rather than promotes the evolution of cooperation.     

Learning in a game context: strategy choice by some keeps learning from evolving in others

Behavioural decisions in a social context commonly have frequency-dependent outcomes and so require analysis using evolutionary game theory. Learning provides a mechanism for tracking changing conditions and it has frequently been predicted to supplant fixed behaviour in shifting environments; yet few studies have examined the evolution of learning specifically in a game-theoretic context. We present a model that examines the evolution of learning in a frequency-dependent context created by a producer–scrounger game, where producers search for their own resources and scroungers usurp the discoveries of producers. We ask whether a learning mutant that can optimize its use of producer and scrounger to local conditions can invade a population of non-learning individuals that play producer and scrounger with fixed probabilities. We find that learning provides an initial advantage but never evolves to fixation. Once a stable equilibrium is attained, the population is always made up of a majority of fixed players and a minority of learning individuals. This result is robust to variation in the initial proportion of fixed individuals, the rate of within- and between-generation environmental change, and population size. Such learning polymorphisms will manifest themselves in a wide range of contexts, providing an important element leading to behavioural syndromes.     

Evolutionary dynamics in structured populations

Evolutionary dynamics shape the living world around us. At the centre of every evolutionary process is a population of reproducing individuals. The structure of that population affects evolutionary dynamics. The individuals can be molecules, cells, viruses, multicellular organisms or humans. Whenever the fitness of individuals depends on the relative abundance of phenotypes in the population, we are in the realm of evolutionary game theory. Evolutionary game theory is a general approach that can describe the competition of species in an ecosystem, the interaction between hosts and parasites, between viruses and cells, and also the spread of ideas and behaviours in the human population. In this perspective, we review the recent advances in evolutionary game dynamics with a particular emphasis on stochastic approaches in finite sized and structured populations. We give simple, fundamental laws that determine how natural selection chooses between competing strategies. We study the well-mixed population, evolutionary graph theory, games in phenotype space and evolutionary set theory. We apply these results to the evolution of cooperation. The mechanism that leads to the evolution of cooperation in these settings could be called ‘spatial selection’: cooperators prevail against defectors by clustering in physical or other spaces.     

Estimating the hidden learning representations

Successful adaptation relies on the ability to learn the consequence of our actions in different environments. However, understanding the neural bases of this ability still represents one of the great challenges of system neuroscience. In fact, the neuronal plasticity changes occurring during learning cannot be fully controlled experimentally and their evolution is hidden. Our approach is to provide hypotheses about the structure and dynamics of the hidden plasticity changes using behavioral learning theory. In fact, behavioral models of animal learning provide testable predictions about the hidden learning representations by formalizing their relation with the observables of the experiment (stimuli, actions and outcomes). Thus, we can understand whether and how the predicted learning processes are represented at the neural level by estimating their evolution and correlating them with neural data. Here, we present a bayesian model approach to estimate the evolution of the internal learning representations from the observations of the experiment (state estimation), and to identify the set of models' parameters (parameter estimation) and the class of behavioral model (model selection) that are most likely to have generated a given sequence of actions and outcomes. More precisely, we use Sequential Monte Carlo methods for state estimation and the maximum likelihood principle (MLP) for model selection and parameter estimation. We show that the method recovers simulated trajectories of learning sessions on a single-trial basis and provides predictions about the activity of different categories of neurons that should participate in the learning process. By correlating the estimated evolutions of the learning variables, we will be able to test the validity of different models of instrumental learning and possibly identify the neural bases of learning.     

The evolution of payoff matrices: providing incentives to cooperate

Most of the work in evolutionary game theory starts with a model of a social situation that gives rise to a particular payoff matrix and analyses how behaviour evolves through natural selection. Here, we invert this approach and ask, given a model of how individuals behave, how the payoff matrix will evolve through natural selection. In particular, we ask whether a prisoner's dilemma game is stable against invasions by mutant genotypes that alter the payoffs. To answer this question, we develop a two-tiered framework with goal-oriented dynamics at the behavioural time scale and a diploid population genetic model at the evolutionary time scale. Our results are two-fold: first, we show that the prisoner's dilemma is subject to invasions by mutants that provide incentives for cooperation to their partners, and that the resulting game is a coordination game similar to the hawk-dove game. Second, we find that for a large class of mutants and symmetric games, a stable genetic polymorphism will exist in the locus determining the payoff matrix, resulting in a complex pattern of behavioural diversity in the population. Our results highlight the importance of considering the evolution of payoff matrices to understand the evolution of animal social systems.     

Why animals lie: how dishonesty and belief can coexist in a signaling system

We develop and apply a simple model for animal communication in which signalers can use a nontrivial frequency of deception without causing listeners to completely lose belief. This common feature of animal communication has been difficult to explain as a stable adaptive outcome of the options and payoffs intrinsic to signaling interactions. Our theory is based on two realistic assumptions. (1) Signals are "overheard" by several listeners or listener types with different payoffs. The signaler may then benefit from using incomplete honesty to elicit different responses from different listener types, such as attracting potential mates while simultaneously deterring competitors. (2) Signaler and listener strategies change dynamically in response to current payoffs for different behaviors. The dynamic equations can be interpreted as describing learning and behavior change by individuals or evolution across generations. We explain how our dynamic model differs from other solution concepts from classical and evolutionary game theory and how it relates to general models for frequency-dependent phenotype dynamics. We illustrate the theory with several applications where deceptive signaling occurs readily in our framework, including bluffing competitors for potential mates or territories. We suggest future theoretical directions to make the models more general and propose some possible experimental tests.     

Independence and interdependence in the nest-site choice by honeybee swarms: Agent-based models, analytical approaches and pattern formation

In a recent paper List, Elsholtz and Seeley (List et al., 2009) have devised an agent-based model of the nest-choice dynamics in swarms of honeybees, and have concluded that both interdependence and independence are needed for the bees to reach a consensus on the best nest site. We here present a simplified version of the model which can be treated analytically with the tools of statistical physics and which largely has the same features as the original dynamics. Based on our analytical approaches it is possible to characterize the co-ordination outcome exactly on the deterministic level, and to a good approximation if stochastic effects are taken into account, reducing the need for computer simulations on the agent-based level. In the second part of the paper we present a spatial extension, and show that transient non-trivial patterns emerge, before consensus is reached. Approaches in terms of Langevin equations for continuous field variables are discussed.     

Evolutionary games on cycles

Traditional evolutionary game theory explores frequency-dependent selection in well-mixed populations without spatial or stochastic effects. But recently there has been much interest in studying the evolutionary game dynamics in spatial settings, on lattices and other graphs. Here, we present an analytic approach for the stochastic evolutionary game dynamics on the simplest possible graph, the cycle. For three different update rules, called 'birth-death' (BD), 'death-birth' (DB) and 'imitation' (IM), we derive exact conditions for natural selection to favour one strategy over another. As specific examples, we consider a coordination game and Prisoner's Dilemma. In the latter case, selection can favour cooperators over defectors for DB and IM updating. We also study the case where the replacement graph of evolutionary updating remains a cycle, but the interaction graph for playing the game is a complete graph. In this setting, all three update rules lead to identical conditions in the limit of weak selection, where we find the '1/3-law' of well-mixed populations.     

Evolution of cooperation under -person snowdrift games

In the animal world, performing a given task which is beneficial to an entire group requires the cooperation of several individuals of that group who often share the workload required to perform the task. The mathematical framework to study the dynamics of collective action is game theory. Here we study the evolutionary dynamics of cooperators and defectors in a population in which groups of individuals engage in N-person, non-excludable public goods games. We explore an N-person generalization of the well-known two-person snowdrift game. We discuss both the case of infinite and finite populations, taking explicitly into consideration the possible existence of a threshold above which collective action is materialized. Whereas in infinite populations, an N-person snowdrift game (NSG) leads to a stable coexistence between cooperators and defectors, the introduction of a threshold leads to the appearance of a new interior fixed point associated with a coordination threshold. The fingerprints of the stable and unstable interior fixed points still affect the evolutionary dynamics in finite populations, despite evolution leading the population inexorably to a monomorphic end-state. However, when the group size and population size become comparable, we find that spite sets in, rendering cooperation unfeasible.     

Aggregation as an antipredator strategy in the rock-paper-scissors model

We study a nonhierarchical tritrophic system, whose predator-prey interactions are described by the rock-paper-scissors game rules. In our stochastic simulations, individuals may move strategically towards the direction with more conspecifics to form clumps instead of moving aimlessly on the lattice. Considering that the conditioning to move gregariously depends on the organism's physical and cognitive abilities, we introduce a maximum distance an individual can perceive the environment and a minimum conditioning level to perform the gregarious movement. We investigate the pattern formation and compute the average size of the single-species spatial domains emerging from the grouping behaviour. The results reveal that the defence tactic reduces the predation risk significantly, being more profitable if individuals perceive further distances, thus creating bigger groups. Our outcomes show that the species with more conditioned organisms dominate the cyclic spatial game, controlling most of the territory. On the other hand, the species with fewer individuals ready to perform aggregation strategy gives its predator the chance to fill the more significant fraction of the grid. The spatial interactions assumed in our numerical experiments constitute a data set that may help biologists and data scientists understand how local interactions influence ecosystem dynamics.     

Co-evolution of learning complexity and social foraging strategies

Variation in learning abilities within populations suggests that complex learning may not necessarily be more adaptive than simple learning. Yet, the high cost of complex learning cannot fully explain this variation without some understanding of why complex learning is too costly for some individuals but not for others. Here we propose that different social foraging strategies can favor different learning strategies (that learn the environment with high or low resolution), thereby maintaining variable learning abilities within populations. Using a genetic algorithm in an agent-based evolutionary simulation of a social foraging game (the producer-scrounger game) we demonstrate how an association evolves between a strategy based on independent search for food (playing a producer) and a complex (high resolution) learning rule, while a strategy that combines independent search and following others (playing a scrounger) evolves an association with a simple (low resolution) learning rule. The reason for these associations is that for complex learning to have an advantage, a large number of learning steps, normally not achieved by scroungers, are necessary. These results offer a general explanation for persistent variation in cognitive abilities that is based on co-evolution of learning rules and social foraging strategies.     

Evolutionary stability and quasi-stationary strategy in stochastic evolutionary game dynamics

Stochastic evolutionary game dynamics for finite populations has recently been widely explored in the study of evolutionary game theory. It is known from the work of Traulsen et al. [2005. Phys. Rev. Lett. 95, 238701] that the stochastic evolutionary dynamics approaches the deterministic replicator dynamics in the limit of large population size. However, sometimes the limiting behavior predicted by the stochastic evolutionary dynamics is not quite in agreement with the steady-state behavior of the replicator dynamics. This paradox inspired us to give reasonable explanations of the traditional concept of evolutionarily stable strategy (ESS) in the context of finite populations. A quasi-stationary analysis of the stochastic evolutionary game dynamics is put forward in this study and we present a new concept of quasi-stationary strategy (QSS) for large but finite populations. It is shown that the consistency between the QSS and the ESS implies that the long-term behavior of the replicator dynamics can be predicted by the quasi-stationary behavior of the stochastic dynamics. We relate the paradox to the time scales and find that the contradiction occurs only when the fixation time scale is much longer than the quasi-stationary time scale. Our work may shed light on understanding the relationship between the deterministic and stochastic methods of modeling evolutionary game dynamics.     

Repeated games and direct reciprocity under active linking

Direct reciprocity relies on repeated encounters between the same two individuals. Here we examine the evolution of cooperation under direct reciprocity in dynamically structured populations. Individuals occupy the vertices of a graph, undergoing repeated interactions with their partners via the edges of the graph. Unlike the traditional approach to evolutionary game theory, where individuals meet at random and have no control over the frequency or duration of interactions, we consider a model in which individuals differ in the rate at which they seek new interactions. Moreover, once a link between two individuals has formed, the productivity of this link is evaluated. Links can be broken off at different rates. Whenever the active dynamics of links is sufficiently fast, population structure leads to a simple transformation of the payoff matrix, effectively changing the game under consideration, and hence paving the way for reciprocators to dominate defectors. We derive analytical conditions for evolutionary stability.     

DeepCME: A deep learning framework for computing solution statistics of the chemical master equation

Stochastic models of biomolecular reaction networks are commonly employed in systems and synthetic biology to study the effects of stochastic fluctuations emanating from reactions involving species with low copy-numbers. For such models, the Kolmogorov’s forward equation is called the chemical master equation (CME), and it is a fundamental system of linear ordinary differential equations (ODEs) that describes the evolution of the probability distribution of the random state-vector representing the copy-numbers of all the reacting species. The size of this system is given by the number of states that are accessible by the chemical system, and for most examples of interest this number is either very large or infinite. Moreover, approximations that reduce the size of the system by retaining only a finite number of important chemical states (e.g. those with non-negligible probability) result in high-dimensional ODE systems, even when the number of reacting species is small. Consequently, accurate numerical solution of the CME is very challenging, despite the linear nature of the underlying ODEs. One often resorts to estimating the solutions via computationally intensive stochastic simulations. The goal of the present paper is to develop a novel deep-learning approach for computing solution statistics of high-dimensional CMEs by reformulating the stochastic dynamics using Kolmogorov’s backward equation. The proposed method leverages superior approximation properties of Deep Neural Networks (DNNs) to reliably estimate expectations under the CME solution for several user-defined functions of the state-vector. This method is algorithmically based on reinforcement learning and it only requires a moderate number of stochastic simulations (in comparison to typical simulation-based approaches) to train the “policy function”. This allows not just the numerical approximation of various expectations for the CME solution but also of its sensitivities with respect to all the reaction network parameters (e.g. rate constants). We provide four examples to illustrate our methodology and provide several directions for future research.     

Analytical Results for Individual and Group Selection of Any Intensity

The idea of evolutionary game theory is to relate the payoff of a game to reproductive success (= fitness). An underlying assumption in most models is that fitness is a linear function of the payoff. For stochastic evolutionary dynamics in finite populations, this leads to analytical results in the limit of weak selection, where the game has a small effect on overall fitness. But this linear function makes the analysis of strong selection difficult. Here, we show that analytical results can be obtained for any intensity of selection, if fitness is defined as an exponential function of payoff. This approach also works for group selection (= multi-level selection). We discuss the difference between our approach and that of inclusive fitness theory.     

MATHEMATICS OF KIN‐ AND GROUP‐SELECTION: FORMALLY EQUIVALENT?

Evolutionary game theory is a general mathematical framework that describes the evolution of social traits. This framework forms the basis of many multilevel selection models and is also frequently used to model evolutionary dynamics on networks. Kin selection, which was initially restricted to describe social interactions between relatives, has also led to a broader mathematical approach, inclusive fitness, that can not only describe social evolution among relatives, but also in group structured populations or on social networks. It turns out that the underlying mathematics of game theory is fundamentally different from the approach of inclusive fitness. Thus, both approaches—evolutionary game theory and inclusive fitness—can be helpful to understand the evolution of social traits in group structured or spatially extended populations.     

Modelling population dynamics in a unicellular social organism community using a minimal model and evolutionary game theory

Most unicellular organisms live in communities and express different phenotypes. Many efforts have been made to study the population dynamics of such complex communities of cells, coexisting as well-coordinated units. Minimal models based on ordinary differential equations are powerful tools that can help us understand complex phenomena. They represent an appropriate compromise between complexity and tractability; they allow a profound and comprehensive analysis, which is still easy to understand. Evolutionary game theory is another powerful tool that can help us understand the costs and benefits of the decision a particular cell of a unicellular social organism takes when faced with the challenges of the biotic and abiotic environment. This work is a binocular view at the population dynamics of such a community through the objectives of minimal modelling and evolutionary game theory. We test the behaviour of the community of a unicellular social organism at three levels of antibiotic stress. Even in the absence of the antibiotic, spikes in the fraction of resistant cells can be observed indicating the importance of bet hedging. At moderate level of antibiotic stress, we witness cyclic dynamics reminiscent of the renowned rock–paper–scissors game. At a very high level, the resistant type of strategy is the most favourable.     

Partner choice creates fairness in humans

Many studies demonstrate that partner choice has played an important role in the evolution of human cooperation, but little work has tested its impact on the evolution of human fairness. In experiments involving divisions of money, people become either over-generous or over-selfish when they are in competition to be chosen as cooperative partners. Hence, it is difficult to see how partner choice could result in the evolution of fair, equal divisions. Here, we show that this puzzle can be solved if we consider the outside options on which partner choice operates. We conduct a behavioural experiment, run agent-based simulations and analyse a game-theoretic model to understand how outside options affect partner choice and fairness. All support the conclusion that partner choice leads to fairness only when individuals have equal outside options. We discuss how this condition has been met in our evolutionary history, and the implications of these findings for our understanding of other aspects of fairness less specific than preferences for equal divisions of resources.