Emergence and resilience of cooperation in the spatial prisoner's dilemma via a reward mechanism

We study the problem of the emergence of cooperation in the spatial Prisoner's Dilemma. The pioneering work by Nowak and May [1992. Evolutionary games and spatial chaos. Nature 415, 424-426] showed that large initial populations of cooperators can survive and sustain cooperation in a square lattice with imitate-the-best evolutionary dynamics. We revisit this problem in a cost-benefit formulation suitable for a number of biological applications. We show that if a fixed-amount reward is established for cooperators to share, a single cooperator can invade a population of defectors and form structures that are resilient to re-invasion even if the reward mechanism is turned off. We discuss analytically the case of the invasion by a single cooperator and present agent-based simulations for small initial fractions of cooperators. Large cooperation levels, in the sustainability range, are found. In the conclusions we discuss possible applications of this model as well as its connections with other mechanisms proposed to promote the emergence of cooperation.     

A microbial modified prisoner's dilemma game: how frequency-dependent selection can lead to random phase variation

Random phase variation (RPV) is a control strategy in which the expression of a cell state or phenotype randomly alternates between discrete 'on' and 'off' states. Though this mode of control is common for bacterial virulence factors like pili and toxins, precise conditions under which RPV confers an advantage have not been well defined. In Part I of this study, we predicted that fluctuating environments select for RPV if transitions between different selective environments cannot be reliably sensed (J. Theor. Biol. (2005)). However, selective forces both inside and outside of human hosts are also likely to be frequency dependent in the sense that the fitnesses of some bacterial states are greatest when rare. Here we show that RPV at slow rates can provide a survival advantage in such a frequency-dependent environment by generating population heterogeneity, essentially mimicking a polymorphism. More surprisingly, RPV at a faster 'optimal' rate can shift the population composition toward an optimal growth rate that exceeds that possible for polymorphic populations, but this optimal strategy is not evolutionarily stable. The population would be most fit if all cells randomly phase varied at the optimal rate, but individual cells have a growth-rate incentive to defect (mutate) to other switching rates or non-phase variable phenotype expression, leading to an overall loss of fitness of the individual and the population. This scenario describes a modified Prisoner's Dilemma game (Evolution and the Theory of Games, Cambridge University Press, Cambridge, New York, 1982, viii, 224pp.; Nature 398 (6726) (1999) 367), with random phase variation at optimal switching rates serving as the cooperation strategy.     

Natural selection of cooperation and degree hierarchy in heterogeneous populations

One of the current theoretical challenges to the explanatory powers of Evolutionary Theory is the understanding of the observed evolutionary survival of cooperative behavior when selfish actions provide higher fitness (reproductive success). In unstructured populations natural selection drives cooperation to extinction. However, when individuals are allowed to interact only with their neighbors, specified by a graph of social contacts, cooperation-promoting mechanisms (known as lattice reciprocity) offer to cooperation the opportunity of evolutionary survival. Recent numerical works on the evolution of Prisoner's Dilemma in complex network settings have revealed that graph heterogeneity dramatically enhances the lattice reciprocity. Here we show that in highly heterogeneous populations, under the graph analog of replicator dynamics, the fixation of a strategy in the whole population is in general an impossible event, for there is an asymptotic partition of the population in three subsets, two in which fixation of cooperation or defection has been reached and a third one which experiences cycles of invasion by the competing strategies. We show how the dynamical partition correlates with connectivity classes and characterize the temporal fluctuations of the fluctuating set, unveiling the mechanisms stabilizing cooperation in macroscopic scale-free structures.     

A simple learning strategy that realizes robust cooperation better than Pavlov in Iterated Prisoners' Dilemma

Pavlov was proposed as a leading strategy for realizing cooperation because it dominates over a long period in evolutionary computer simulations of the Iterated Prisoners' Dilemma. However, our numerical calculations reveal that Pavlov and also any other cooperative strategy are not evolutionarily stable among all stochastic strategies with memory of only one previous move. We propose simple learning based on reinforcement. The learner changes its internal state, depending on an evaluation of whether the score in the previous round is larger than a critical value (aspiration level), which is genetically fixed. The current internal state decides the learner's move, but we found that the aspiration level determines its final behavior. The cooperative variant, having an intermediate aspiration level, is not an evolutionarily stable strategy (ESS) when evaluation is binary (good or bad). However, when the evaluation is quantified some cooperative variants can invade not only All-C, Tit-For-Tat (TFT), and Pavlov but also noncooperative variants with different aspiration levels. Moreover, they establish robust cooperation, which is evolutionarily stable against invasion by All-C, All-D, TFT, Pavlov, and noncooperative variants, and they receive a high score even when the error rate is high. Our results suggest that mutual cooperation can be maintained when players have a primitive learning ability. Received: June 23, 2000 / Accepted: October 1, 2000     

Self-deception as a mediator of cooperation and defection in varying social contexts described in the iterated prisoner's dilemma

Deception and self-deception may be two related psychological capacities serving to enhance an individual's inclusive fitness. Self-deception has been defined as any psychological act in which one thought or belief is held at the expense of another. This study was designed to test the notion that we may deceive ourselves about our own intentions or the intentions of others in order to facilitate cooperation, if the cooperative behavior has the potential to improve fitness. We may also deceive ourselves in situations where we may need to compete with others, if not competing might represent a loss of fitness. The Self-Deceptio Questionnaire was administered to 80 women and 70 men to determine their levels of self-deception. A series of vignettes, conforming to the Prisoner's Dilemma game format, were administered in questionnaire material to measure participant's predispositions to cooperate or defect in three different contexts (family, mating, and neutral). Men predicted they would be less cooperative than women over all conditions, and participants anticipated cooperating more with kin than with non-kin. Moreover, participants who were high self-deceivers chose more cooperative responses in both family and mating contexts than those with low self-deception scores. Finally, the three-way interaction among sex, context, and level of self-deception suggested that the role of selfdeception in cooperation may vary according to the sex of the individual and the context.     

Evolution of interactions and cooperation in the spatial prisoner's dilemma game

We study the evolution of cooperation in the spatial prisoner's dilemma game where players are allowed to establish new interactions with others. By employing a simple coevolutionary rule entailing only two crucial parameters, we find that different selection criteria for the new interaction partners as well as their number vitally affect the outcome of the game. The resolution of the social dilemma is most probable if the selection favors more successful players and if their maximally attainable number is restricted. While the preferential selection of the best players promotes cooperation irrespective of game parametrization, the optimal number of new interactions depends somewhat on the temptation to defect. Our findings reveal that the "making of new friends" may be an important activity for the successful evolution of cooperation, but also that partners must be selected carefully and their number limited.     

Evolution of indirect reciprocity in groups of various sizes and comparison with direct reciprocity

Recently many studies have investigated the evolution of indirect reciprocity through which cooperative action is returned by a third individual, e.g. individual A helped B and then receives help from C. Most studies on indirect reciprocity have presumed that only two individuals take part in a single interaction (group), e.g. A helps B and C helps A. In this paper, we investigate the evolution of indirect reciprocity when more than two individuals take part in a single group, and compare the result with direct reciprocity through which cooperative action is directly returned by the recipient. Our analyses show the following. In the population with discriminating cooperators and unconditional defectors, whether implementation error is included or not, (i) both strategies are evolutionarily stable and the evolution of indirect reciprocity becomes more difficult as group size increases, and (ii) the condition for the evolution of indirect reciprocity under standing reputation criterion where the third individuals distinguish between justified and unjustified defections is more relaxed than that under image scoring reputation criterion in which the third individuals do not distinguish with. Furthermore, in the population that also includes unconditional cooperators, (iii) in the presence of errors in implementation, the discriminating strategy is evolutionarily stable not only under standing but also under image scoring if group size is larger than two. Finally, (iv) in the absence of errors in implementation, the condition for the evolution of direct reciprocity is equivalent to that for the evolution of indirect reciprocity under standing, and, in the presence of errors, the condition for the evolution of direct reciprocity is very close to that for the evolution of indirect reciprocity under image scoring.     

The evolution of punishment and apology: an iterated prisoner's dilemma model

Recently, behaviors that seem to function as punishment or apology have been reported among non-human primates as well as humans. Such behaviors appear to play an important role in maintaining cooperation between individuals. Therefore, the evolution of these behaviors should be examined from the viewpoint of the evolution of cooperation. The iterated prisoner's dilemma (IPD) game is generally considered to be a standard model for the evolution of cooperation. In the present study, strategies accompanied by punishment-like attacks or apology-like behavior were introduced into the common IPD simulation. Punishment and apology were represented by the P signal and the AS signal given immediately after defection. A strategy with the P and AS signals, named the pPAS strategy, was proved to be an evolutionarily stable strategy under certain conditions. Numerical simulations were carried out according to different assigned values of the costs of punishment and apology. The simulations showed that pPAS could dominate the population (1) when the cost of giving P is relatively small, (2) when the cost of receiving P is relatively large, or (3) when the cost of giving AS is relatively large. The relative cost of giving AS had the clearest effect on the success of pPAS. pPAS can dominate the population even when a dominance asymmetry of the costs between two players was introduced. The present results suggest the possible evolution of social behaviors like punishment or apology as a means of maintaining cooperation. This revised version was published online in November 2006 with corrections to the Cover Date.     

Resolving the iterated prisoner's dilemma: theory and reality

Pairs of unrelated individuals face a prisoner's dilemma if cooperation is the best mutual outcome, but each player does best to defect regardless of his partner's behaviour. Although mutual defection is the only evolutionarily stable strategy in one-shot games, cooperative solutions based on reciprocity can emerge in iterated games. Among the most prominent theoretical solutions are the so-called bookkeeping strategies, such as tit-for-tat, where individuals copy their partner's behaviour in the previous round. However, the lack of empirical data conforming to predicted strategies has prompted the suggestion that the iterated prisoner's dilemma (IPD) is neither a useful nor realistic basis for investigating cooperation. Here, we discuss several recent studies where authors have used the IPD framework to interpret their data. We evaluate the validity of their approach and highlight the diversity of proposed solutions. Strategies based on precise accounting are relatively uncommon, perhaps because the full set of assumptions of the IPD model are rarely satisfied. Instead, animals use a diverse array of strategies that apparently promote cooperation, despite the temptation to cheat. These include both positive and negative reciprocity, as well as long-term mutual investments based on 'friendships'. Although there are various gaps in these studies that remain to be filled, we argue that in most cases, individuals could theoretically benefit from cheating and that cooperation cannot therefore be explained with the concept of positive pseudo-reciprocity. We suggest that by incorporating empirical data into the theoretical framework, we may gain fundamental new insights into the evolution of mutual reciprocal investment in nature.     

Evolution of contingent altruism when cooperation is expensive

The ubiquity of cooperation has motivated a major research program over the last 50 years to discover ever more minimal conditions for the evolution of altruism. One important line of work is based on favoritism toward those who appear to be close relatives. Another important line is based on continuing interactions, whether between individuals (e.g., reciprocity) or between lines of descent in a viscous population. Here, we use an agent-based model to demonstrate a new mechanism that combines both lines of work to show when and how favoritism toward apparently similar others can evolve in the first place. The mechanism is the joint operation of viscosity and of tags (heritable, observable, and initially arbitrary characteristics), which serve as weak and potentially deceptive indicators of relatedness. Although tags are insufficient to support cooperation alone, we show that this joint mechanism vastly increases the range of environments in which contingent altruism can evolve in viscous populations. Even though our model is quite simple, the subtle dynamics underlying our results are not tractable using formal analytic tools (such as analysis of evolutionarily stable strategies), but are amenable to agent-based simulation.     

Spatial invasion of cooperation

The evolutionary puzzle of cooperation describes situations where cooperators provide a fitness benefit to other individuals at some cost to themselves. Under Darwinian selection, the evolution of cooperation is a conundrum, whereas non-cooperation (or defection) is not. In the absence of supporting mechanisms, cooperators perform poorly and decrease in abundance. Evolutionary game theory provides a powerful mathematical framework to address the problem of cooperation using the prisoner's dilemma. One well-studied possibility to maintain cooperation is to consider structured populations, where each individual interacts only with a limited subset of the population. This enables cooperators to form clusters such that they are more likely to interact with other cooperators instead of being exploited by defectors. Here we present a detailed analysis of how a few cooperators invade and expand in a world of defectors. If the invasion succeeds, the expansion process takes place in two stages: first, cooperators and defectors quickly establish a local equilibrium and then they uniformly expand in space. The second stage provides good estimates for the global equilibrium frequencies of cooperators and defectors. Under hospitable conditions, cooperators typically form a single, ever growing cluster interspersed with specks of defectors, whereas under more hostile conditions, cooperators form isolated, compact clusters that minimize exploitation by defectors. We provide the first quantitative assessment of the way cooperators arrange in space during invasion and find that the macroscopic properties and the emerging spatial patterns reveal information about the characteristics of the underlying microscopic interactions.     

Synergy and discounting of cooperation in social dilemmas

The emergence and maintenance of cooperation by natural selection is an enduring conundrum in evolutionary biology, which has been studied using a variety of game theoretical models inspired by different biological situations. The most widely studied games are the Prisoner's Dilemma, the Snowdrift game and by-product mutualism for pairwise interactions, as well as Public Goods games in larger groups of interacting individuals. Here, we present a general framework for cooperation in social dilemmas in which all the traditional scenarios can be recovered as special cases. In social dilemmas, cooperators provide a benefit to the group at some cost, while defectors exploit the group by reaping the benefits without bearing the costs of cooperation. Using the concepts of discounting and synergy for describing how benefits accumulate when more than one cooperator is present in a group of interacting individuals, we recover the four basic scenarios of evolutionary dynamics given by (i) dominating defection, (ii) coexistence of defectors and cooperators, (iii) dominating cooperation and (iv) bi-stability, in which cooperators and defectors cannot invade each other. Generically, for groups of three or more interacting individuals further, more complex, dynamics can occur. Our framework provides the first unifying approach to model cooperation in different kinds of social dilemmas.     

The continuous public goods game and the evolution of cooperative sex ratios

Some animals, such as Melittobia wasps and surface-living mites, have extremely female-biased sex ratios that cannot be explained by the existing local mate competition (LMC) theories. The restricted production of sons may entail cooperation among mothers, enabling the production of more daughters and avoiding severe competition among sons for insemination access. These unusual examples are characterized by the long-term cohabitation of egg-layers (foundresses) on resource patches and possible contact with oviposited eggs. By applying the logic of mutual policing, we develop a novel game theoretical model for the evolution of cooperation in sex-ratio traits. This is the first attempt to model the evolution of sex ratios based on iterated games. We assumed that foundresses have two abilities to enable mutual policing: they can monitor the sex ratio in the resource patch, and they can punish defectors that produce an overabundance of males. Numerical analysis and evolutionary simulations demonstrate that cooperative low sex ratios can evolve when the number of foundresses per patch is sufficiently small. Our model predicts a slight, but steady increase in oviposition sex ratios with an increase in the number of foundresses, which mimics the phenomenon observed in several animals with extremely female-biased sex ratios. We also discuss the relationship between our model and other models of sex-ratio evolution.     

Strong reciprocity, human cooperation, and the enforcement of social norms

This paper provides strong evidence challenging the self-interest assumption that dominates the behavioral sciences and much evolutionary thinking. The evidence indicates that many people have a tendency to voluntarily cooperate, if treated fairly, and to punish noncooperators. We call this behavioral propensity “strong reciprocity” and show empirically that it can lead to almost universal cooperation in circumstances in which purely self-interested behavior would cause a complete breakdown of cooperation. In addition, we show that people are willing to punish those who behaved unfairly towards a third person or who defected in a Prisoner’s Dilemma game with a third person. This suggests that strong reciprocity is a powerful device for the enforcement of social norms involving, for example, food sharing or collective action. Strong reciprocity cannot be rationalized as an adaptive trait by the leading evolutionary theories of human cooperation (in other words, kin selection, reciprocal altruism, indirect reciprocity, and costly signaling theory). However, multilevel selection theories of cultural evolution are consistent with strong reciprocity.     

Anti-social punishment can prevent the co-evolution of punishment and cooperation

The evolution of cooperation is one of the great puzzles in evolutionary biology. Punishment has been suggested as one solution to this problem. Here punishment is generally defined as incurring a cost to inflict harm on a wrong-doer. In the presence of punishers, cooperators can gain higher payoffs than non-cooperators. Therefore cooperation may evolve as long as punishment is prevalent in the population. Theoretical models have revealed that spatial structure can favor the co-evolution of punishment and cooperation, by allowing individuals to only play and compete with those in their immediate neighborhood. However, those models have usually assumed that punishment is always targeted at non-cooperators. In light of recent empirical evidence of punishment targeted at cooperators, we relax this assumption and study the effect of so-called ‘anti-social punishment’. We find that evolution can favor anti-social punishment, and that when anti-social punishment is possible costly punishment no longer promotes cooperation. As there is no reason to assume that cooperators cannot be the target of punishment during evolution, our results demonstrate serious restrictions on the ability of costly punishment to allow the evolution of cooperation in spatially structured populations. Our results also help to make sense of the empirical observation that defectors will sometimes pay to punish cooperators.     

Revisiting “The Evolution of Reciprocity in Sizable Groups”: Continuous reciprocity in the repeated n-person prisoner's dilemma

For many years in evolutionary science, the consensus view has been that while reciprocal altruism can evolve in dyadic interactions, it is unlikely to evolve in sizable groups. This view had been based on studies which have assumed cooperation to be discrete rather than continuous (i.e., individuals can either fully cooperate or else fully defect, but they cannot continuously vary their level of cooperation). In real world cooperation, however, cooperation is often continuous. In this paper, we re-examine the evolution of reciprocity in sizable groups by presenting a model of the n-person prisoner's dilemma that assumes continuous rather than discrete cooperation. This model shows that continuous reciprocity has a dramatically wider basin of attraction than discrete reciprocity, and that this basin's size increases with efficiency of cooperation (marginal per capita return). Further, we find that assortative interaction interacts synergistically with continuous reciprocity to a much greater extent than it does with discrete reciprocity. These results suggest that previous models may have underestimated reciprocity's adaptiveness in groups. However, we also find that the invasion of continuous reciprocators into a population of unconditional defectors becomes realistic only within a narrow parameter space in which the efficiency of cooperation is close to its maximum bound. Therefore our model suggests that continuous reciprocity can evolve in large groups more easily than discrete reciprocity only under unusual circumstances.     

Evolution of cooperation: cooperation defeats defection in the cornfield model

"Cooperation" defines any behavior that enhances the fitness of a group (e.g. a community or species), but which, by its nature, can be exploited by selfish individuals, meaning, firstly, that selfish individuals derive an advantage from exploitation which is greater than the average advantage that accrues to unselfish individuals. Secondly, exploitation has no intrinsic fitness value except in the presence of the "cooperative behavior". The mathematics is described by the simple Prisoner's Dilemma Game (PDG). It has previously been shown that koinophilia (the avoidance of sexual mates displaying unusual or atypical phenotypic features, such as mutations) stabilizes any inherited strategy in the simple or iterated PDG, meaning that it cannot be displaced by rare forms of alternative behavior which arise through mutation or occasional migration. In the present model equal numbers of cooperators and defectors (in the simple PDG) were randomly spread in a two-dimensional "cornfield" with uniformly distributed resources. Every individual was koinophilic, and interacted (sexually and in the PDG tournaments) only with individuals from within its immediate neighborhood. This model therefore tested whether cooperation can outcompete defection or selfishness in a straight, initially equally matched, evolutionary battle. The results show that in the absence of koinophilia cooperation was rapidly driven to extinction. With koinophilia there was a very rapid loss of cooperators in the first few generations, but thereafter cooperation slowly spread, ultimately eliminating defection completely. This result was critically dependent on sampling effects of neighborhoods. Small samples (resulting from low population densities or small neighborhood sizes) increase the probability that a chance neighborhood comes to consist predominantly of cooperators. A sexual preference for the most common phenotype in the neighborhood then makes that phenotype more common still. Once this occurs cooperation's spread becomes almost inevitable.     

The paradox of cooperation benefits

It seems obvious that as the benefits of cooperation increase, the share of cooperators in the population should also increase. It is well known that positive assortment between cooperative types, for instance in spatially structured populations, provide better conditions for the evolution of cooperation than complete mixing. This study demonstrates that, assuming positive assortment, under most conditions higher cooperation benefits also increase the share of cooperators. On the other hand, under a specified range of payoff values, when at least two payoff parameters are modified, the reverse is true. The conditions for this paradox are determined for two-person social dilemmas: the Prisoner's Dilemma, the Hawks and Doves game, and the Stag Hunt game, assuming global selection and positive assortment.     

Eco-evolutionary feedback and the invasion of cooperation in prisoner's dilemma games

Unveiling the origin and forms of cooperation in nature poses profound challenges in evolutionary ecology. The prisoner's dilemma game is an important metaphor for studying the evolution of cooperation. We here classified potential mechanisms for cooperation evolution into schemes of frequency- and density-dependent selection, and focused on the density-dependent selection in the ecological prisoner's dilemma games. We found that, although assortative encounter is still the necessary condition in ecological games for cooperation evolution, a harsh environment, indicated by a high mortality, can foster the invasion of cooperation. The Hamilton rule provides a fundamental condition for the evolution of cooperation by ensuring an enhanced relatedness between players in low-density populations. Incorporating ecological dynamics into evolutionary games opens up a much wider window for the evolution of cooperation, and exhibits a variety of complex behaviors of dynamics, such as limit and heteroclinic cycles. An alternative evolutionary, or rather succession, sequence was proposed that cooperation first appears in harsh environments, followed by the invasion of defection, which leads to a common catastrophe. The rise of cooperation (and altruism), thus, could be much easier in the density-dependent ecological games than in the classic frequency-dependent evolutionary games.     

Effects of encounter in a population of spatial prisoner’s dilemma players

We study the evolution of cooperation in spatial prisoner’s dilemma games, whereby each player extends its interaction scope by trying to interact with a certain number of encounters randomly chosen from its non-neighbors, in addition to its permanently linked nearest neighbors. Furthermore, the non-neighbors treat the initiative interactions in two scenarios: definitely accepting that from the cooperators, whereas guardedly interacting with defectors with an acceptance probability which may take arbitrary value in [0,1]. Importantly, our results reveal that the proposed encounter mechanism is a potent extrinsic factor that is able to boost cooperation when appropriately adjusting the values of the encounter number and acceptance probability, though rational players would always defect in one-shot encounters, regardless of the action from the counterparts. We hope our studies may help understand that the proposed encounter mechanism is also an important ingredient of a flourishing cooperative society.