Stochastic payoff evaluation increases the temperature of selection

We study stochastic evolutionary game dynamics in populations of finite size. Moreover, each individual has a randomly distributed number of interactions with other individuals. Therefore, the payoff of two individuals using the same strategy can be different. The resulting "payoff stochasticity" reduces the intensity of selection and therefore increases the temperature of selection. A simple mean-field approximation is derived that captures the average effect of the payoff stochasticity. Correction terms to the mean-field theory are computed and discussed.     

Mutation-selection equilibrium in games with multiple strategies

In evolutionary games the fitness of individuals is not constant but depends on the relative abundance of the various strategies in the population. Here we study general games among n strategies in populations of large but finite size. We explore stochastic evolutionary dynamics under weak selection, but for any mutation rate. We analyze the frequency dependent Moran process in well-mixed populations, but almost identical results are found for the Wright-Fisher and Pairwise Comparison processes. Surprisingly simple conditions specify whether a strategy is more abundant on average than 1/n, or than another strategy, in the mutation-selection equilibrium. We find one condition that holds for low mutation rate and another condition that holds for high mutation rate. A linear combination of these two conditions holds for any mutation rate. Our results allow a complete characterization of n×n games in the limit of weak selection.     

Mutation-selection equilibrium in games with mixed strategies

We develop a new method for studying stochastic evolutionary game dynamics of mixed strategies. We consider the general situation: there are n pure strategies whose interactions are described by an n×n payoff matrix. Players can use mixed strategies, which are given by the vector (p₁,…,p_n). Each entry specifies the probability to use the corresponding pure strategy. The sum over all entries is one. Therefore, a mixed strategy is a point in the simplex S_n. We study evolutionary dynamics in a well-mixed population of finite size. Individuals reproduce proportional to payoff. We consider the case of weak selection, which means the payoff from the game is only a small contribution to overall fitness. Reproduction can be subject to mutation; a mutant adopts a randomly chosen mixed strategy. We calculate the average abundance of every mixed strategy in the stationary distribution of the mutation-selection process. We find the crucial conditions that specify if a strategy is favored or opposed by selection. One condition holds for low mutation rate, another for high mutation rate. The result for any mutation rate is a linear combination of those two. As a specific example we study the Hawk-Dove game. We prove general statements about the relationship between games with pure and with mixed strategies.     

Analytical Results for Individual and Group Selection of Any Intensity

The idea of evolutionary game theory is to relate the payoff of a game to reproductive success (= fitness). An underlying assumption in most models is that fitness is a linear function of the payoff. For stochastic evolutionary dynamics in finite populations, this leads to analytical results in the limit of weak selection, where the game has a small effect on overall fitness. But this linear function makes the analysis of strong selection difficult. Here, we show that analytical results can be obtained for any intensity of selection, if fitness is defined as an exponential function of payoff. This approach also works for group selection (= multi-level selection). We discuss the difference between our approach and that of inclusive fitness theory.     

The probability of fixation of a single mutant in an exchangeable selection model

The Cannings exchangeable model for a finite population in discrete time is extended to incorporate selection. The probability of fixation of a mutant type is studied under the assumption of weak selection. An exact formula for the derivative of this probability with respect to the intensity of selection is deduced, and developed in the case of a single mutant. This formula is expressed in terms of mean coalescence times under neutrality assuming that the coefficient of selection for the mutant type has a derivative with respect to the intensity of selection that takes a polynomial form with respect to the frequency of the mutant type. An approximation is obtained in the case where this derivative is a continuous function of the mutant frequency and the population size is large. This approximation is consistent with a diffusion approximation under moment conditions on the number of descendants of a single individual in one time step. Applications to evolutionary game theory in finite populations are presented.      

Learning to reach by reinforcement learning using a receptive field based function approximation approach with continuous actions

Reinforcement learning methods can be used in robotics applications especially for specific target-oriented problems, for example the reward-based recalibration of goal directed actions. To this end still relatively large and continuous state-action spaces need to be efficiently handled. The goal of this paper is, thus, to develop a novel, rather simple method which uses reinforcement learning with function approximation in conjunction with different reward-strategies for solving such problems. For the testing of our method, we use a four degree-of-freedom reaching problem in 3D-space simulated by a two-joint robot arm system with two DOF each. Function approximation is based on 4D, overlapping kernels (receptive fields) and the state-action space contains about 10,000 of these. Different types of reward structures are being compared, for example, reward-on- touching-only against reward-on-approach. Furthermore, forbidden joint configurations are punished. A continuous action space is used. In spite of a rather large number of states and the continuous action space these reward/punishment strategies allow the system to find a good solution usually within about 20 trials. The efficiency of our method demonstrated in this test scenario suggests that it might be possible to use it on a real robot for problems where mixed rewards can be defined in situations where other types of learning might be difficult. This work was supported by EU-Grant PACO-PLUS.     

Cooperation enhanced by the ‘survival of the fittest’ rule in prisoner's dilemma games on complex networks

Prevalence of cooperation within groups of selfish individuals is puzzling in that it contradicts with the basic premise of natural selection, whereby we introduce a model of strategy evolution taking place on evolving networks based on Darwinian ‘survival of the fittest’ rule. In the present work, players whose payoffs are below a certain threshold will be deleted and the same number of new nodes will be added to the network to maintain the constant system size. Furthermore, the networking effect is also studied via implementing simulations on four typical network structures. Numerical results show that cooperators can obtain the biggest boost if the elimination threshold is fine-tuned. Notably, this coevolutionary rule drives the initial networks to evolve into statistically stationary states with a broad-scale degree distribution. Our results may provide many more insights for understanding the coevolution of strategy and network topology under the mechanism of nature selection whereby superior individuals will prosper and inferior ones be eliminated.     

Emergence and resilience of cooperation in the spatial prisoner's dilemma via a reward mechanism

We study the problem of the emergence of cooperation in the spatial Prisoner's Dilemma. The pioneering work by Nowak and May [1992. Evolutionary games and spatial chaos. Nature 415, 424-426] showed that large initial populations of cooperators can survive and sustain cooperation in a square lattice with imitate-the-best evolutionary dynamics. We revisit this problem in a cost-benefit formulation suitable for a number of biological applications. We show that if a fixed-amount reward is established for cooperators to share, a single cooperator can invade a population of defectors and form structures that are resilient to re-invasion even if the reward mechanism is turned off. We discuss analytically the case of the invasion by a single cooperator and present agent-based simulations for small initial fractions of cooperators. Large cooperation levels, in the sustainability range, are found. In the conclusions we discuss possible applications of this model as well as its connections with other mechanisms proposed to promote the emergence of cooperation.     

Effects of encounter in a population of spatial prisoner’s dilemma players

We study the evolution of cooperation in spatial prisoner’s dilemma games, whereby each player extends its interaction scope by trying to interact with a certain number of encounters randomly chosen from its non-neighbors, in addition to its permanently linked nearest neighbors. Furthermore, the non-neighbors treat the initiative interactions in two scenarios: definitely accepting that from the cooperators, whereas guardedly interacting with defectors with an acceptance probability which may take arbitrary value in [0,1]. Importantly, our results reveal that the proposed encounter mechanism is a potent extrinsic factor that is able to boost cooperation when appropriately adjusting the values of the encounter number and acceptance probability, though rational players would always defect in one-shot encounters, regardless of the action from the counterparts. We hope our studies may help understand that the proposed encounter mechanism is also an important ingredient of a flourishing cooperative society.     

Competition for resources can explain patterns of social and individual learning in nature

In nature, animals often ignore socially available information despite the multiple theoretical benefits of social learning over individual trial-and-error learning. Using information filtered by others is quicker, more efficient and less risky than randomly sampling the environment. To explain the mix of social and individual learning used by animals in nature, most models penalize the quality of socially derived information as either out of date, of poor fidelity or costly to acquire. Competition for limited resources, a fundamental evolutionary force, provides a compelling, yet hitherto overlooked, explanation for the evolution of mixed-learning strategies. We present a novel model of social learning that incorporates competition and demonstrates that (i) social learning is favoured when competition is weak, but (ii) if competition is strong social learning is favoured only when resource quality is highly variable and there is low environmental turnover. The frequency of social learning in our model always evolves until it reduces the mean foraging success of the population. The results of our model are consistent with empirical studies showing that individuals rely less on social information where resources vary little in quality and where there is high within-patch competition. Our model provides a framework for understanding the evolution of social learning, a prerequisite for human cumulative culture.     

The replicator equation on graphs

We study evolutionary games on graphs. Each player is represented by a vertex of the graph. The edges denote who meets whom. A player can use any one of n strategies. Players obtain a payoff from interaction with all their immediate neighbors. We consider three different update rules, called 'birth-death', 'death-birth' and 'imitation'. A fourth update rule, 'pairwise comparison', is shown to be equivalent to birth-death updating in our model. We use pair approximation to describe the evolutionary game dynamics on regular graphs of degree k. In the limit of weak selection, we can derive a differential equation which describes how the average frequency of each strategy on the graph changes over time. Remarkably, this equation is a replicator equation with a transformed payoff matrix. Therefore, moving a game from a well-mixed population (the complete graph) onto a regular graph simply results in a transformation of the payoff matrix. The new payoff matrix is the sum of the original payoff matrix plus another matrix, which describes the local competition of strategies. We discuss the application of our theory to four particular examples, the Prisoner's Dilemma, the Snow-Drift game, a coordination game and the Rock-Scissors-Paper game.     

Evolutionary graph theory: breaking the symmetry between interaction and replacement

We study evolutionary dynamics in a population whose structure is given by two graphs: the interaction graph determines who plays with whom in an evolutionary game; the replacement graph specifies the geometry of evolutionary competition and updating. First, we calculate the fixation probabilities of frequency dependent selection between two strategies or phenotypes. We consider three different update mechanisms: birth-death, death-birth and imitation. Then, as a particular example, we explore the evolution of cooperation. Suppose the interaction graph is a regular graph of degree h, the replacement graph is a regular graph of degree g and the overlap between the two graphs is a regular graph of degree l. We show that cooperation is favored by natural selection if b/c>hg/l. Here, b and c denote the benefit and cost of the altruistic act. This result holds for death-birth updating, weak-selection and large population size. Note that the optimum population structure for cooperators is given by maximum overlap between the interaction and the replacement graph (g=h=l), which means that the two graphs are identical. We also prove that a modified replicator equation can describe how the expected values of the frequencies of an arbitrary number of strategies change on replacement and interaction graphs: the two graphs induce a transformation of the payoff matrix.     

Evolutionary stability on graphs

Evolutionary stability is a fundamental concept in evolutionary game theory. A strategy is called an evolutionarily stable strategy (ESS), if its monomorphic population rejects the invasion of any other mutant strategy. Recent studies have revealed that population structure can considerably affect evolutionary dynamics. Here we derive the conditions of evolutionary stability for games on graphs. We obtain analytical conditions for regular graphs of degree k>2. Those theoretical predictions are compared with computer simulations for random regular graphs and for lattices. We study three different update rules: birth-death (BD), death-birth (DB), and imitation (IM) updating. Evolutionary stability on sparse graphs does not imply evolutionary stability in a well-mixed population, nor vice versa. We provide a geometrical interpretation of the ESS condition on graphs.     

Evolutionary stability and quasi-stationary strategy in stochastic evolutionary game dynamics

Stochastic evolutionary game dynamics for finite populations has recently been widely explored in the study of evolutionary game theory. It is known from the work of Traulsen et al. [2005. Phys. Rev. Lett. 95, 238701] that the stochastic evolutionary dynamics approaches the deterministic replicator dynamics in the limit of large population size. However, sometimes the limiting behavior predicted by the stochastic evolutionary dynamics is not quite in agreement with the steady-state behavior of the replicator dynamics. This paradox inspired us to give reasonable explanations of the traditional concept of evolutionarily stable strategy (ESS) in the context of finite populations. A quasi-stationary analysis of the stochastic evolutionary game dynamics is put forward in this study and we present a new concept of quasi-stationary strategy (QSS) for large but finite populations. It is shown that the consistency between the QSS and the ESS implies that the long-term behavior of the replicator dynamics can be predicted by the quasi-stationary behavior of the stochastic dynamics. We relate the paradox to the time scales and find that the contradiction occurs only when the fixation time scale is much longer than the quasi-stationary time scale. Our work may shed light on understanding the relationship between the deterministic and stochastic methods of modeling evolutionary game dynamics.     

Evolution of cooperation under -person snowdrift games

In the animal world, performing a given task which is beneficial to an entire group requires the cooperation of several individuals of that group who often share the workload required to perform the task. The mathematical framework to study the dynamics of collective action is game theory. Here we study the evolutionary dynamics of cooperators and defectors in a population in which groups of individuals engage in N-person, non-excludable public goods games. We explore an N-person generalization of the well-known two-person snowdrift game. We discuss both the case of infinite and finite populations, taking explicitly into consideration the possible existence of a threshold above which collective action is materialized. Whereas in infinite populations, an N-person snowdrift game (NSG) leads to a stable coexistence between cooperators and defectors, the introduction of a threshold leads to the appearance of a new interior fixed point associated with a coordination threshold. The fingerprints of the stable and unstable interior fixed points still affect the evolutionary dynamics in finite populations, despite evolution leading the population inexorably to a monomorphic end-state. However, when the group size and population size become comparable, we find that spite sets in, rendering cooperation unfeasible.     

How inconsistency between attitude and behavior persists through cultural transmission

Individuals tend to conform their behavior to that of the majority. Consequently, an individual's behavior is not always consistent with his or her attitude, and such inconsistency sometimes causes mental distress. Understanding the mechanism of sustaining inconsistency between attitude and behavior is a challenging problem from the viewpoint of evolutionary theory. We constructed an evolutionary game theory model in which each player has an attitude and behavior toward a single social norm, and the players' attitudes and behaviors are affected by three types of cultural transmission: vertical, oblique, and horizontal. We assumed that strategy is a combination of attitude and behavior and that the process of learning or transmitting the social norm depends on the life stage of each player. Adults play a coordination game in which players whose behaviors match those of the majority obtain a high payoff, which is diminished by any inconsistency between attitude and behavior. The adults' strategies are passed to newborns via vertical transmission, and the frequency of a newborn's replication of strategy is proportional to the corresponding adult's payoff. Newborns imitate behaviors of unrelated adults via oblique transmission. Juveniles change their attitudes or behaviors by observing other juveniles' behaviors or inferring other juveniles' attitudes (horizontal transmission). We conclude that the key factor for sustaining inconsistency between attitude and behavior is the ability of players to infer and imitate others' attitudes, and that oblique transmission promotes inconsistency.