Evolutionary dynamics of the continuous iterated prisoner's dilemma

The iterated prisoner's dilemma (IPD) has been widely used in the biological and social sciences to model dyadic cooperation. While most of this work has focused on the discrete prisoner's dilemma, in which actors choose between cooperation and defection, there has been some analysis of the continuous IPD, in which actors can choose any level of cooperation from zero to one. Here, we analyse a model of the continuous IPD with a limited strategy set, and show that a generous strategy achieves the maximum possible payoff against its own type. While this strategy is stable in a neighborhood of the equilibrium point, the equilibrium point itself is always vulnerable to invasion by uncooperative strategies, and hence subject to eventual destabilization. The presence of noise or errors has no effect on this result. Instead, generosity is favored because of its role in increasing contributions to the most efficient level, rather than in counteracting the corrosiveness of noise. Computer simulation using a single-locus infinite alleles Gaussian mutation model suggest that outcomes ranging from a stable cooperative polymorphism to complete collapse of cooperation are possible depending on the magnitude of the mutational variance. Also, making the cost of helping a convex function of the amount of help provided makes it more difficult for cooperative strategies to invade a non-cooperative equilibrium, and for the cooperative equilibrium to resist destabilization by non-cooperative strategies. Finally, we demonstrate that a much greater degree of assortment is required to destabilize a non-cooperative equilibrium in the continuous IPD than in the discrete IPD. The continuous model outlined here suggests that incremental amounts of cooperation lead to rapid decay of cooperation and thus even a large degree of assortment will not be sufficient to allow cooperation to increase when cooperators are rare. The extreme degree of assortment required to destabilize the non-cooperative equilibrium, as well as the instability of the cooperative equilibrium, may help explain why cooperation in Prisoner's Dilemmas is so rare in nature.     

Resolving the iterated prisoner's dilemma: theory and reality

Pairs of unrelated individuals face a prisoner's dilemma if cooperation is the best mutual outcome, but each player does best to defect regardless of his partner's behaviour. Although mutual defection is the only evolutionarily stable strategy in one-shot games, cooperative solutions based on reciprocity can emerge in iterated games. Among the most prominent theoretical solutions are the so-called bookkeeping strategies, such as tit-for-tat, where individuals copy their partner's behaviour in the previous round. However, the lack of empirical data conforming to predicted strategies has prompted the suggestion that the iterated prisoner's dilemma (IPD) is neither a useful nor realistic basis for investigating cooperation. Here, we discuss several recent studies where authors have used the IPD framework to interpret their data. We evaluate the validity of their approach and highlight the diversity of proposed solutions. Strategies based on precise accounting are relatively uncommon, perhaps because the full set of assumptions of the IPD model are rarely satisfied. Instead, animals use a diverse array of strategies that apparently promote cooperation, despite the temptation to cheat. These include both positive and negative reciprocity, as well as long-term mutual investments based on 'friendships'. Although there are various gaps in these studies that remain to be filled, we argue that in most cases, individuals could theoretically benefit from cheating and that cooperation cannot therefore be explained with the concept of positive pseudo-reciprocity. We suggest that by incorporating empirical data into the theoretical framework, we may gain fundamental new insights into the evolution of mutual reciprocal investment in nature.     

Perfect reciprocity is the only evolutionarily stable strategy in the continuous iterated prisoner's dilemma

Theoretical studies have shown that cooperation tends to evolve when interacting individuals have positively correlated phenotypes. In the present article, we explore the situation where this correlation results from information exchange between social partners, and behavioral flexibility. We consider the game 'continuous iterated prisoner's dilemma'. The level of cooperation expressed by individuals in this game, together with their ability to respond to one another, both evolve as two aspects of their behavioral strategy. The conditions for a strategy to be evolutionarily stable in this game are degenerate, and earlier works were thus unable to find a single ESS. However, a detailed invasion analysis, together with the study of evolution in finite populations, reveals that natural selection favors strategies whereby individuals respond to their opponent's actions in a perfectly mirrored (i.e., correlated) fashion. As a corollary, the overall payoff of social interactions (i.e., the amount of cooperation) is maximized because couples of correlated partners effectively become the units of selection.     

Individual heterogeneity and costly punishment: a volunteer's dilemma

Social control and the enforcement of social norms glue a society together. It has been shown theoretically and empirically that informal punishment of wrongdoers fosters cooperation in human groups. Most of this research has focused on voluntary and uncoordinated punishment carried out by individual group members. However, as punishment is costly, it is an open question as to why humans engage in the punishment of wrongdoers even in one-time-only encounters. While evolved punitive preferences have been advocated as proximate explanations for such behaviour, the strategic nature of the punishment situation has remained underexplored. It has been suggested to conceive of the punishment situation as a volunteer''s dilemma (VOD), where only one individual''s action is necessary and sufficient to punish the wrongdoer. Here, we show experimentally that implementing the punishment situation as a VOD sustains cooperation in an environment where punishers and non-punishers coexist. Moreover, we show that punishment-cost heterogeneity allows individuals to tacitly agree on only the strongest group member carrying out the punishment, thereby increasing the effectiveness and efficiency of social norm enforcement. Our results corroborate that costly peer punishment can be explained without assuming punitive preferences and show that centralized sanctioning institutions can emerge from arbitrary individual differences.     

The evolution of punishment through reputation

Punishment of non-cooperators has been observed to promote cooperation. Such punishment is an evolutionary puzzle because it is costly to the punisher while beneficial to others, for example, through increased social cohesion. Recent studies have concluded that punishing strategies usually pay less than some non-punishing strategies. These findings suggest that punishment could not have directly evolved to promote cooperation. However, while it is well established that reputation plays a key role in human cooperation, the simple threat from a reputation of being a punisher may not have been sufficiently explored yet in order to explain the evolution of costly punishment. Here, we first show analytically that punishment can lead to long-term benefits if it influences one''s reputation and thereby makes the punisher more likely to receive help in future interactions. Then, in computer simulations, we incorporate up to 40 more complex strategies that use different kinds of reputations (e.g. from generous actions), or strategies that not only include punitive behaviours directed towards defectors but also towards cooperators for example. Our findings demonstrate that punishment can directly evolve through a simple reputation system. We conclude that reputation is crucial for the evolution of punishment by making a punisher more likely to receive help in future interactions, and that experiments investigating the beneficial effects of punishment in humans should include reputation as an explicit feature.     

Evolving learning rules and emergence of cooperation in spatial prisoner's dilemma

In the evolutionary Prisoner's dilemma (PD) game, agents play with each other and update their strategies in every generation according to some microscopic dynamical rule. In its spatial version, agents do not play with every other but, instead, interact only with their neighbours, thus mimicking the existing of a social or contact network that defines who interacts with whom. In this work, we explore evolutionary, spatial PD systems consisting of two types of agents, each with a certain update (reproduction, learning) rule. We investigate two different scenarios: in the first case, update rules remain fixed for the entire evolution of the system; in the second case, agents update both strategy and update rule in every generation. We show that in a well-mixed population the evolutionary outcome is always full defection. We subsequently focus on two-strategy competition with nearest-neighbour interactions on the contact network and synchronised update of strategies. Our results show that, for an important range of the parameters of the game, the final state of the system is largely different from that arising from the usual setup of a single, fixed dynamical rule. Furthermore, the results are also very different if update rules are fixed or evolve with the strategies. In these respect, we have studied representative update rules, finding that some of them may become extinct while others prevail. We describe the new and rich variety of final outcomes that arise from this co-evolutionary dynamics. We include examples of other neighbourhoods and asynchronous updating that confirm the robustness of our conclusions. Our results pave the way to an evolutionary rationale for modelling social interactions through game theory with a preferred set of update rules.     

The resident's dilemma: a female choice model for the evolution of alternative mating strategies in lekking male ruffs (Philomachus pugnax)

Previous models for the evolution of alternative male matingbehavior have virtually ignored the role of female choice. Wepresent a model in which female choice favors the evolutionand maintenance of alternative mating strategies in male ruffs,Philomachus pugnax. Resident male ruffe establish and defendcourts on leks against other residents, while non-territorialsatellite males move between leks and among courts on a lek.Residents appear to actively recruit satellites to their courts,even though satellites may mate with females once there. Residentbehavior toward satellites and data on female behavior suggestthat residents benefit from a satellite's presence due to somefemale preference for mating on co-occupied courts. However,if all residents accept satellites, none gains any relativeadvantage, yet all pay the costs of having satellites on theircourt. We present a game theoretical model that shows that therelative nature of female choice places residents in an evolutionarydilemma with respect to satellite acceptance. Although all residentswould benefit if satellites could be cooperatively excludedfrom leks, the only evolutionarily stable strategy for individualresidents is to defect and accept satellites. The model alsodemonstrates that this "resident's dilemma" likely exists onlyin a local sense, since the failure of residents to cooperativelyexclude satellites from leks need not result in globally lowerpayoffs, due to frequency-dependent selection on the proportionof satellites in the population. Our analysis suggests thatthe resident-satellite relationship in ruffs, despite its obviouscompetitive elements, is fundamentally a cooperative associationfavored by female choice. Female choice has also been proposedas the primary mechanism selecting for male association to formleks in ruffe. In this context, resident-satellite associationsmay be thought of as transitory "leks within a lek     

The iterated continuous prisoner's dilemma game cannot explain the evolution of interspecific mutualism in unstructured populations

The evolutionionary origin of inter- and intra-specific cooperation among non-related individuals has been a great challenge for biologists for decades. Recently, the continuous prisoner's dilemma game has been introduced to study this problem. In function of previous payoffs, individuals can change their cooperative investment iteratively in this model system. Killingback and Doebeli (Am. Nat. 160 (2002) 421-438) have shown analytically that intra-specific cooperation can emerge in this model system from originally non-cooperating individuals living in a non-structured population. However, it is also known from an earlier numerical work that inter-specific cooperation (mutualism) cannot evolve in a very similar model. The only difference here is that cooperation occurs among individuals of different species. Based on the model framework used by Killingback and Doebeli (2002), this Note proves analytically that mutualism indeed cannot emerge in this model system. Since numerical results have revealed that mutualism can evolve in this model system if individuals interact in a spatially structured manner, our work emphasizes indirectly the role of spatial structure of populations in the origin of mutualism.     

The evolution of altruism: Correlation,cost, and benefit

A simple and general criterion is derived for the evolution of altruism when individuals interact in pairs. It is argued that the treatment of this problem in kin selection theory and in game theory are special cases of this general criterion.My thanks to James Crow, Carter Denniston, Lee Dugarkin, David Wilson, and an anonymous referee of this journal for helpful discussion.     

The economics of altruistic punishment and the maintenance of cooperation

Explaining the evolution and maintenance of cooperation among unrelated individuals is one of the fundamental problems in biology and the social sciences. Recent findings suggest that altruistic punishment is an important mechanism maintaining cooperation among humans. We experimentally explore the boundaries of altruistic punishment to maintain cooperation by varying both the cost and the impact of punishment, using an exceptionally extensive subject pool. Our results show that cooperation is only maintained if conditions for altruistic punishment are relatively favourable: low cost for the punisher and high impact on the punished. Our results indicate that punishment is strongly governed by its cost-to-impact ratio and that its effect on cooperation can be pinned down to one single variable: the threshold level of free-riding that goes unpunished. Additionally, actual pay-offs are the lowest when altruistic punishment maintains cooperation, because the pay-off destroyed through punishment exceeds the gains from increased cooperation. Our results are consistent with the interpretation that punishment decisions come from an amalgam of emotional response and cognitive cost-impact analysis and suggest that altruistic punishment alone can hardly maintain cooperation under multi-level natural selection. Uncovering the workings of altruistic punishment as has been done here is important because it helps predicting under which conditions altruistic punishment is expected to maintain cooperation.     

Wrath of God: religious primes and punishment

Recent evidence indicates that priming participants with religious concepts promotes prosocial sharing behaviour. In the present study, we investigated whether religious priming also promotes the costly punishment of unfair behaviour. A total of 304 participants played a punishment game. Before the punishment stage began, participants were subliminally primed with religion primes, secular punishment primes or control primes. We found that religious primes strongly increased the costly punishment of unfair behaviours for a subset of our participants--those who had previously donated to a religious organization. We discuss two proximate mechanisms potentially underpinning this effect. The first is a 'supernatural watcher' mechanism, whereby religious participants punish unfair behaviours when primed because they sense that not doing so will enrage or disappoint an observing supernatural agent. The second is a 'behavioural priming' mechanism, whereby religious primes activate cultural norms pertaining to fairness and its enforcement and occasion behaviour consistent with those norms. We conclude that our results are consistent with dual inheritance proposals about religion and cooperation, whereby religions harness the byproducts of genetically inherited cognitive mechanisms in ways that enhance the survival prospects of their adherents.     

The economic basis of cooperation: tradeoffs between selfishness and generosity

The current study examined the economics of cooperation in controlled-payoffgames by using captive blue jays, Cyanocitta cristata. Thisinvestigation used a special feeding apparatus to test for thestability of cooperative choice in a series of iterated games.The jays experienced experimentally determined game theoreticalpayoff matrices, which determined the distribution of food tothemselves and their opponent, depending on their decision tocooperate or defect. The experiment tested four game matrices,called the cooperate only, defect only, prisoner's dilemma,and opponent control treatments. This study found little cooperationin the defect only and prisoner's dilemma treatments. Cooperationoccurred significantly more often in the opponent control treatment.These findings suggest that the jays attend to short-term consequences;they do not cooperate in the absence of an immediate benefit(defect only), even if a long-term benefit may exist (prisoner'sdilemma). The opponent control treatment suggests that cooperationcan occur when an individual's benefits depend completely onthe actions of others; therefore, generosity is cheap. Thisstudy, therefore, agrees with recent studies in proposing alternativemodels of cooperation.     

Emergence and resilience of cooperation in the spatial prisoner's dilemma via a reward mechanism

We study the problem of the emergence of cooperation in the spatial Prisoner's Dilemma. The pioneering work by Nowak and May [1992. Evolutionary games and spatial chaos. Nature 415, 424-426] showed that large initial populations of cooperators can survive and sustain cooperation in a square lattice with imitate-the-best evolutionary dynamics. We revisit this problem in a cost-benefit formulation suitable for a number of biological applications. We show that if a fixed-amount reward is established for cooperators to share, a single cooperator can invade a population of defectors and form structures that are resilient to re-invasion even if the reward mechanism is turned off. We discuss analytically the case of the invasion by a single cooperator and present agent-based simulations for small initial fractions of cooperators. Large cooperation levels, in the sustainability range, are found. In the conclusions we discuss possible applications of this model as well as its connections with other mechanisms proposed to promote the emergence of cooperation.     

The evolution of punishment in n-player public goods games: a volunteer's dilemma

The evolution of punishment to stabilize cooperation in n-player games has been treated as a second-order social dilemma, where contributions to punishment of free-riders are altruistic. Hence it may only evolve under highly restricted conditions. Here, we build on recent insights using the volunteer's dilemma as an alternative payoff matrix for the evolution of cooperation. The key feature of a volunteer's dilemma is that the benefits of cooperation are a nonlinear function of the number of contributors, meaning that cooperation is negatively frequency dependent. We propose that nonlinear returns are also an inherent feature of punishment and that this insight allows for a simple and novel explanation of how punishment evolves in groups.     

The continuous prisoner's dilemma and the evolution of cooperation through reciprocal altruism with variable investment

Understanding the evolutionary origin and persistence of cooperative behavior is a fundamental biological problem. The standard "prisoner's dilemma," which is the most widely adopted framework for studying the evolution of cooperation through reciprocal altruism between unrelated individuals, does not allow for varying degrees of cooperation. Here we study the continuous iterated prisoner's dilemma, in which cooperative investments can vary continuously in each round. This game has been previously considered for a class of reactive strategies in which current investments are based on the partner's previous investment. In the standard iterated prisoner's dilemma, such strategies are inferior to strategies that take into account both players' previous moves, as is exemplified by the evolutionary dominance of "Pavlov" over "tit for tat." Consequently, we extend the analysis of the continuous prisoner's dilemma to a class of strategies in which current investments depend on previous payoffs and, hence, on both players' previous investments. We show, both analytically and by simulation, that payoff-based strategies, which embody the intuitively appealing idea that individuals invest more in cooperative interactions when they profit from these interactions, provide a natural explanation for the gradual evolution of cooperation from an initially noncooperative state and for the maintenance of cooperation thereafter.     

The evolution of strong reciprocity: cooperation in heterogeneous populations

How do human groups maintain a high level of cooperation despite a low level of genetic relatedness among group members? We suggest that many humans have a predisposition to punish those who violate group-beneficial norms, even when this imposes a fitness cost on the punisher. Such altruistic punishment is widely observed to sustain high levels of cooperation in behavioral experiments and in natural settings. We offer a model of cooperation and punishment that we call STRONG RECIPROCITY: where members of a group benefit from mutual adherence to a social norm, strong reciprocators obey the norm and punish its violators, even though as a result they receive lower payoffs than other group members, such as selfish agents who violate the norm and do not punish, and pure cooperators who adhere to the norm but free-ride by never punishing. Our agent-based simulations show that, under assumptions approximating likely human environments over the 100000 years prior to the domestication of animals and plants, the proliferation of strong reciprocators when initially rare is highly likely, and that substantial frequencies of all three behavioral types can be sustained in a population. As a result, high levels of cooperation are sustained. Our results do not require that group members be related or that group extinctions occur.     

High strength-of-ties and low mobility enable the evolution of third-party punishment

As punishment can be essential to cooperation and norm maintenance but costly to the punisher, many evolutionary game-theoretic studies have explored how direct punishment can evolve in populations. Compared to direct punishment, in which an agent acts to punish another for an interaction in which both parties were involved, the evolution of third-party punishment (3PP) is even more puzzling, because the punishing agent itself was not involved in the original interaction. Despite significant empirical studies of 3PP, little is known about the conditions under which it can evolve. We find that punishment reputation is not, by itself, sufficient for the evolution of 3PP. Drawing on research streams in sociology and psychology, we implement a structured population model and show that high strength-of-ties and low mobility are critical for the evolution of responsible 3PP. Only in such settings of high social-structural constraint are punishers able to induce self-interested agents toward cooperation, making responsible 3PP ultimately beneficial to individuals as well as the collective. Our results illuminate the conditions under which 3PP is evolutionarily adaptive in populations. Responsible 3PP can evolve and induce cooperation in cases where other mechanisms alone fail to do so.     

False friends are worse than bitter enemies: “Altruistic” punishment of in-group members

One of the most critical features of human society is the pervasiveness of cooperation in social and economic exchanges. Moreover, social scientists have found overwhelming evidence that such cooperative behavior is likely to be directed toward in-group members. We propose that the group-based nature of cooperation includes punishment behavior. Punishment behavior is used to maintain cooperation within systems of social exchange and, thus, is directed towards members of an exchange system. Because social exchanges often take place within groups, we predict that punishment behavior is used to maintain cooperation in the punisher's group. Specifically, punishment behavior is directed toward in-group members who are found to be noncooperators. To examine this, we conducted a gift-giving game experiment with third-party punishment. The results of the experiment (N=90) support the following hypothesis: Participants who are cooperative in a gift-giving game punish noncooperative in-group members more severely than they punish noncooperative out-group members.     

The evolution of altruism: game theory in multilevel selection and inclusive fitness

Although the prisoner's dilemma (PD) has been used extensively to study reciprocal altruism, here we show that the n-player prisoner's dilemma (NPD) is also central to two other prominent theories of the evolution of altruism: inclusive fitness and multilevel selection. An NPD model captures the essential factors for the evolution of altruism directly in its parameters and integrates important aspects of these two theories such as Hamilton's rule, Simpson's paradox, and the Price covariance equation. The model also suggests a simple interpretation of the Price selection decomposition and an alternative decomposition that is symmetrical and complementary to it. In some situations this alternative shows the temporal changes in within- and between-group selection more clearly than the Price equation. In addition, we provide a new perspective on strong vs. weak altruism by identifying their different underlying game structures (based on absolute fitness) and showing how their evolutionary dynamics are nevertheless similar under selection (based on relative fitness). In contrast to conventional wisdom, the model shows that both strong and weak altruism can evolve in periodically formed random groups of non-conditional strategies if groups are multigenerational. An integrative approach based on the NPD helps unify different perspectives on the evolution of altruism.     

Revisiting “The Evolution of Reciprocity in Sizable Groups”: Continuous reciprocity in the repeated n-person prisoner's dilemma

For many years in evolutionary science, the consensus view has been that while reciprocal altruism can evolve in dyadic interactions, it is unlikely to evolve in sizable groups. This view had been based on studies which have assumed cooperation to be discrete rather than continuous (i.e., individuals can either fully cooperate or else fully defect, but they cannot continuously vary their level of cooperation). In real world cooperation, however, cooperation is often continuous. In this paper, we re-examine the evolution of reciprocity in sizable groups by presenting a model of the n-person prisoner's dilemma that assumes continuous rather than discrete cooperation. This model shows that continuous reciprocity has a dramatically wider basin of attraction than discrete reciprocity, and that this basin's size increases with efficiency of cooperation (marginal per capita return). Further, we find that assortative interaction interacts synergistically with continuous reciprocity to a much greater extent than it does with discrete reciprocity. These results suggest that previous models may have underestimated reciprocity's adaptiveness in groups. However, we also find that the invasion of continuous reciprocators into a population of unconditional defectors becomes realistic only within a narrow parameter space in which the efficiency of cooperation is close to its maximum bound. Therefore our model suggests that continuous reciprocity can evolve in large groups more easily than discrete reciprocity only under unusual circumstances.