Models of cooperation based on the Prisoner's Dilemma and the Snowdrift game

Understanding the mechanisms that can lead to the evolution of cooperation through natural selection is a core problem in biology. Among the various attempts at constructing a theory of cooperation, game theory has played a central role. Here, we review models of cooperation that are based on two simple games: the Prisoner's Dilemma, and the Snowdrift game. Both games are two‐person games with two strategies, to cooperate and to defect, and both games are social dilemmas. In social dilemmas, cooperation is prone to exploitation by defectors, and the average payoff in populations at evolutionary equilibrium is lower than it would be in populations consisting of only cooperators. The difference between the games is that cooperation is not maintained in the Prisoner's Dilemma, but persists in the Snowdrift game at an intermediate frequency. As a consequence, insights gained from studying extensions of the two games differ substantially. We review the most salient results obtained from extensions such as iteration, spatial structure, continuously variable cooperative investments, and multi‐person interactions. Bridging the gap between theoretical and empirical research is one of the main challenges for future studies of cooperation, and we conclude by pointing out a number of promising natural systems in which the theory can be tested experimentally.     

Norway rats reciprocate help according to the quality of help they received

Direct reciprocity, according to the decision rule ‘help someone who has helped you before’, reflects cooperation based on the principle of postponed benefits. A predominant factor influencing Homo sapiens'' motivation to reciprocate is an individ­ual''s perceived benefit resulting from the value of received help. But hitherto it has been unclear whether other species also base their decision to cooperate on the quality of received help. Previous experiments have demonstrated that Norway rats, Rattus norvegicus, cooperate using direct reciprocity decision rules in a variant of the iterated Prisoner''s Dilemma, where they preferentially help cooperators instead of defectors. But, as the quality of obtained benefits has not been varied, it is yet unclear whether rats use the value of received help as decision criterion to pay help back. Here, we tested whether rats distinguish between different cooperators depending purely on the quality of their help. Our data show that a rat''s propensity to reciprocate help is, indeed, adjusted to the perceived quality of the partner''s previous help. When cooperating with two conspecific partners expending the same effort, rats apparently rely on obtained benefit to adjust their level of returned help.     

A Theoretical Analysis of Temporal Difference Learning in the Iterated Prisoner’s Dilemma Game

Direct reciprocity is a chief mechanism of mutual cooperation in social dilemma. Agents cooperate if future interactions with the same opponents are highly likely. Direct reciprocity has been explored mostly by evolutionary game theory based on natural selection. Our daily experience tells, however, that real social agents including humans learn to cooperate based on experience. In this paper, we analyze a reinforcement learning model called temporal difference learning and study its performance in the iterated Prisoner’s Dilemma game. Temporal difference learning is unique among a variety of learning models in that it inherently aims at increasing future payoffs, not immediate ones. It also has a neural basis. We analytically and numerically show that learners with only two internal states properly learn to cooperate with retaliatory players and to defect against unconditional cooperators and defectors. Four-state learners are more capable of achieving a high payoff against various opponents. Moreover, we numerically show that four-state learners can learn to establish mutual cooperation for sufficiently small learning rates.     

Cooperation beyond the dyad: on simple models and a complex society

Players in Axelrod and Hamilton''s model of cooperation were not only in a Prisoner''s Dilemma, but by definition, they were also trapped in a dyad. But animals are rarely so restricted and even the option to interact with third parties allows individuals to escape from the Prisoner''s Dilemma into a much more interesting and varied world of cooperation, from the apparently rare ‘parcelling’ to the widespread phenomenon of market effects. Our understanding of by-product mutualism, pseudo-reciprocity and the snowdrift game is also enriched by thinking ‘beyond the dyad’. The concepts of by-product mutualism and pseudo-reciprocity force us to think again about our basic definitions of cooperative behaviour (behaviour by a single individual) and cooperation (the outcome of an interaction between two or more individuals). Reciprocity is surprisingly rare outside of humans, even among large-brained ‘intelligent’ birds and mammals. Are humans unique in having extensive cooperative interactions among non-kin and an integrated cognitive system for mediating reciprocity? Perhaps, but our best chance for finding a similar phenomenon may be in delphinids, which also live in large societies with extensive cooperative interactions among non-relatives. A system of nested male alliances in bottlenose dolphins illustrates the potential and difficulties of finding a complex system of cooperation close to our own.     

Suspending disbelief—of Wynne-Edwards and his reception

Critique of Wynne-Edwards' views on population regulation and sociality suppose a population of discrete, mutually exclusive groups essential to his thought. Yet both his past and present work focus on continually distributed, philopatric populations; his critics have argued the untenability of a position never his own. Wynne-Edwardsian ‘group selection’ focuses on local population productivity under philopatry. A ‘group’ is a local confluence of genotypes which need not be reified, and group selection consists of the differential replication (hence heritability) of the local social environment in which a genotype is embedded. Differential productivity contingent on social environment can eliminate some relational structures on genotypes in favor of others, creating an expanding wave of population productivity as in Wright's shifting balance metaphor. Such a process is inherent in the evolution of reciprocity, where cooperators must cluster to successfully invade a population of defectors. Regulation of resource exploitation in continuously distributed populations may be modeled as overlapping n-person Prisoner's Dilemmas, where each individual participates in several distinct commons and defection represents local over-exploitation of resources.     

Evolution of cooperation in patchy habitat under patch decay and isolation

The spatial version of Prisoners Dilemma (PD) is studied, which incorporates habitat decay through change in the mortality parameter and habitat isolation through change in the colonization coefficient. We found four kinds of evolutionary results, which are affected profoundly by the elements of the payoff matrix and the ratio of the colonization coefficient to the mortality parameter: population extinction, a pure cooperator population, coexistence of cooperators and defectors, and a pure defector population. First, the parameter region of cooperation (pure cooperator and coexistence region) shrinks with an increase in the cooperative cost, and that of defection extends. The increase in cooperative reward makes the cooperative region extend and the defector region become small. Second, the cooperative reward can compensate for the extinction risk due to habitat destruction and allow a population to survive even if the colonization coefficient is smaller than the mortality parameter. Third, although habitat destruction (including decay and isolation) increase the extinction risk of a population, moderate external power can push the evolution of cooperation ahead of population extinction, and even make a completely cooperative world come into being. Finally, for certain values of elements of the payoff matrix, the population suffering habitat destruction can maintain a stable population size by regulating the frequencies of cooperators and defectors. This implies that the multi-behavior strategy within a population may be a mechanism to defend against the influences of a changing environment.     

Evolutionary dynamics of continuous strategy games on graphs and social networks under weak selection

Understanding the emergence of cooperation among selfish individuals has been a long-standing puzzle, which has been studied by a variety of game models. Most previous studies presumed that interactions between individuals are discrete, but it seems unrealistic in real systems. Recently, there are increasing interests in studying game models with a continuous strategy space. Existing research work on continuous strategy games mainly focuses on well-mixed populations. Especially, little theoretical work has been conducted on their evolutionary dynamics in a structured population. In the previous work (Zhong et al., BioSystems, 2012), we showed that under strong selection, continuous and discrete strategies have significantly different equilibrium and game dynamics in spatially structured populations. In this paper, we further study evolutionary dynamics of continuous strategy games under weak selection in structured populations. By using the fixation probability based stochastic dynamics, we derive exact conditions of natural selection favoring cooperation for the death–birth updating scheme. We also present a network gain decomposition of the game equilibrium, which might provide a new view of the network reciprocity in a quantitative way. Finally, we make a detailed comparison between games using discrete and continuous strategies. As compared to the former, we find that for the latter (i) the same selection conditions are derived for the general 2 × 2 game; especially, the rule b/c > k in a simplified Prisoner's Dilemma is valid as well; however, (ii) for a coordination game, interestingly, the risk-dominant strategy is disfavored. Numerical simulations have also been conducted to validate our results.     

A simple scaling of the effectiveness of supporting mutual cooperation in donor-recipient games by various reciprocity mechanisms

In donor-recipient games (DRG), one of the sub-classes of Prisoner's Dilemma (PD), it is well-known that a game structure is described by two parameters benefit (b) and cost (c) of cooperation. By means of a series of numerical experiments, we proved that the effectiveness of supporting mutual cooperation in DRG by various reciprocity mechanisms can be expressed in a single game structural parameter, b/c. This also implies that the dilemma strength in various donor-recipient games with various reciprocity mechanisms can be evaluated only by b/c, which is consistent with the previous novel finding by Nowak. It was also discussed whether this kind of parameterization idea can be extended to general games in PD game class.     

The interplay of cognition and cooperation

Cooperation often involves behaviours that reduce immediate payoffs for actors. Delayed benefits have often been argued to pose problems for the evolution of cooperation because learning such contingencies may be difficult as partners may cheat in return. Therefore, the ability to achieve stable cooperation has often been linked to a species'' cognitive abilities, which is in turn linked to the evolution of increasingly complex central nervous systems. However, in their famous 1981 paper, Axelrod and Hamilton stated that in principle even bacteria could play a tit-for-tat strategy in an iterated Prisoner''s Dilemma. While to our knowledge this has not been documented, interspecific mutualisms are present in bacteria, plants and fungi. Moreover, many species which have evolved large brains in complex social environments lack convincing evidence in favour of reciprocity. What conditions must be fulfilled so that organisms with little to no brainpower, including plants and single-celled organisms, can, on average, gain benefits from interactions with partner species? On the other hand, what conditions favour the evolution of large brains and flexible behaviour, which includes the use of misinformation and so on? These questions are critical, as they begin to address why cognitive complexity would emerge when ‘simple’ cooperation is clearly sufficient in some cases. This paper spans the literature from bacteria to humans in our search for the key variables that link cooperation and deception to cognition.     

Cognitive and Motivational Requirements for the Emergence of Cooperation in a Rat Social Game

Background

Game theory and the Prisoner''s Dilemma (PD) game in particular, which captures the paradox of cooperative interactions that lead to benefits but entail costs to the interacting individuals, have constituted a powerful tool in the study of the mechanisms of reciprocity. However, in non-human animals most tests of reciprocity in PD games have resulted in sustained defection strategies. As a consequence, it has been suggested that under such stringent conditions as the PD game humans alone have evolved the necessary cognitive abilities to engage in reciprocity, namely, numerical discrimination, memory and control of temporal discounting.

Methodology/Principal Findings

We use an iterated PD game to test rats (Rattus norvegicus) for the presence of such cognitive abilities by manipulating the strategy of the opponent, Tit-for-Tat and Pseudo-Random, or the relative size of the temptation to defect. We found that rats shape their behaviour according to the opponent''s strategy and the relative outcome resulting from cooperative or defective moves. Finally, we show that the behaviour of rats is contingent upon their motivational state (hungry versus sated).

Conclusions/Significance

Here we show that rats understand the payoff matrix of the PD game and the strategy of the opponent. Importantly, our findings reveal that rats possess the necessary cognitive capacities for reciprocity-based cooperation to emerge in the context of a prisoner''s dilemma. Finally, the validation of the rat as a model to study reciprocity-based cooperation during the PD game opens new avenues of research in experimental neuroscience.     

Social experiments in the mesoscale: humans playing a spatial prisoner's dilemma

Background

The evolutionary origin of cooperation among unrelated individuals remains a key unsolved issue across several disciplines. Prominent among the several mechanisms proposed to explain how cooperation can emerge is the existence of a population structure that determines the interactions among individuals. Many models have explored analytically and by simulation the effects of such a structure, particularly in the framework of the Prisoner''s Dilemma, but the results of these models largely depend on details such as the type of spatial structure or the evolutionary dynamics. Therefore, experimental work suitably designed to address this question is needed to probe these issues.

Methods and Findings

We have designed an experiment to test the emergence of cooperation when humans play Prisoner''s Dilemma on a network whose size is comparable to that of simulations. We find that the cooperation level declines to an asymptotic state with low but nonzero cooperation. Regarding players'' behavior, we observe that the population is heterogeneous, consisting of a high percentage of defectors, a smaller one of cooperators, and a large group that shares features of the conditional cooperators of public goods games. We propose an agent-based model based on the coexistence of these different strategies that is in good agreement with all the experimental observations.

Conclusions

In our large experimental setup, cooperation was not promoted by the existence of a lattice beyond a residual level (around 20%) typical of public goods experiments. Our findings also indicate that both heterogeneity and a “moody” conditional cooperation strategy, in which the probability of cooperating also depends on the player''s previous action, are required to understand the outcome of the experiment. These results could impact the way game theory on graphs is used to model human interactions in structured groups.     

Effects of increasing the number of players and memory size in the iterated Prisoner's Dilemma: a numerical approach

The Prisoner''s Dilemma has become a paradigm for the evolution of altruistic behaviour. Here we present results of numerical simulations of the infinitely iterated stochastic simultaneous Prisoner''s Dilemma considering players with longer memory, encounters of more than two players as well as different pay-off values. This provides us with a better foundation to compare theoretical results to experimental data. We show that the success of the strategy Pavlov, regardless of its simplicity, is far more general by having an outstanding role in the iterated N-player N-memory Prisoner''s Dilemma. Besides, we study influences of increased memory sizes in the iterated two-player Prisoner''s Dilemma, and present comparisons to results of experiments with first-year students.     

Forgiver Triumphs in Alternating Prisoner's Dilemma

Cooperative behavior, where one individual incurs a cost to help another, is a wide spread phenomenon. Here we study direct reciprocity in the context of the alternating Prisoner''s Dilemma. We consider all strategies that can be implemented by one and two-state automata. We calculate the payoff matrix of all pairwise encounters in the presence of noise. We explore deterministic selection dynamics with and without mutation. Using different error rates and payoff values, we observe convergence to a small number of distinct equilibria. Two of them are uncooperative strict Nash equilibria representing always-defect (ALLD) and Grim. The third equilibrium is mixed and represents a cooperative alliance of several strategies, dominated by a strategy which we call Forgiver. Forgiver cooperates whenever the opponent has cooperated; it defects once when the opponent has defected, but subsequently Forgiver attempts to re-establish cooperation even if the opponent has defected again. Forgiver is not an evolutionarily stable strategy, but the alliance, which it rules, is asymptotically stable. For a wide range of parameter values the most commonly observed outcome is convergence to the mixed equilibrium, dominated by Forgiver. Our results show that although forgiving might incur a short-term loss it can lead to a long-term gain. Forgiveness facilitates stable cooperation in the presence of exploitation and noise.     

ASSORTMENT AND THE EVOLUTION OF GENERALIZED RECIPROCITY

Reciprocity is often invoked to explain cooperation. Reciprocity is cognitively demanding, and both direct and indirect reciprocity require that individuals store information about the propensity of their partners to cooperate. By contrast, generalized reciprocity, wherein individuals help on the condition that they received help previously, only relies on whether an individual received help in a previous encounter. Such anonymous information makes generalized reciprocity hard to evolve in a well‐mixed population, as the strategy will lose out to pure defectors. Here we analyze a model for the evolution of generalized reciprocity, incorporating assortment of encounters, to investigate the conditions under which it will evolve. We show that, in a well‐mixed population, generalized reciprocity cannot evolve. However, incorporating assortment of encounters can favor the evolution of generalized reciprocity in which indiscriminate cooperation and defection are both unstable. We show that generalized reciprocity can evolve under both the prisoner's dilemma and the snowdrift game.     

Long-term social bonds promote cooperation in the iterated Prisoner's Dilemma

Reciprocal altruism, one of the most probable explanations for cooperation among non-kin, has been modelled as a Prisoner''s Dilemma. According to this game, cooperation could evolve when individuals, who expect to play again, use conditional strategies like tit-for-tat or Pavlov. There is evidence that humans use such strategies to achieve mutual cooperation, but most controlled experiments with non-human animals have failed to find cooperation. One reason for this could be that subjects fail to cooperate because they behave as if they were to play only once. To assess this hypothesis, we conducted an experiment with monogamous zebra finches (Taeniopygia guttata) that were tested in a two-choice apparatus, with either their social partner or an experimental opponent of the opposite sex. We found that zebra finches maintained high levels of cooperation in an iterated Prisoner''s Dilemma game only when interacting with their social partner. Although other mechanisms may have contributed to the observed difference between the two treatments, our results support the hypothesis that animals do not systematically give in to the short-term temptation of cheating when long-term benefits exist. Thus, our findings contradict the commonly accepted idea that reciprocal altruism will be rare in non-human animals.     

The continuous Prisoner's dilemma: II. Linear reactive strategies with noise.

We present a general model for the Continuous Prisoner's Dilemma and study the effect of errors. We find that cooperative strategies that can resist invasion by defectors are optimistic (make high initial offers), generous (always offer more cooperation than the partner did in the previous round) and uncompromising (offer full cooperation only if the partner does). A necessary condition for the emergence of cooperation in the continuous Prisoner's Dilemma with noise is b (1-p)>c, where b and c denote, respectively, the benefit and cost of cooperation, while p is the error rate. This relation can be reformulated as an error threshold: cooperation can only emerge if the probability of making a mistake is below a critical value. We note, however, that cooperation in the continuous Prisoner's Dilemma with noise does not seem to be evolutionarily stable: while it is possible to find cooperative strategies that resist invasion by defectors, such cooperators are generally invaded by more cooperative strategies which eventually yield to defectors. Thus, the long-term evolution of the continuous Prisoner's Dilemma is either characterized by unending cycles or by stable polymorphisms of cooperators and defectors.     

Strong reciprocity, human cooperation, and the enforcement of social norms

This paper provides strong evidence challenging the self-interest assumption that dominates the behavioral sciences and much evolutionary thinking. The evidence indicates that many people have a tendency to voluntarily cooperate, if treated fairly, and to punish noncooperators. We call this behavioral propensity “strong reciprocity” and show empirically that it can lead to almost universal cooperation in circumstances in which purely self-interested behavior would cause a complete breakdown of cooperation. In addition, we show that people are willing to punish those who behaved unfairly towards a third person or who defected in a Prisoner’s Dilemma game with a third person. This suggests that strong reciprocity is a powerful device for the enforcement of social norms involving, for example, food sharing or collective action. Strong reciprocity cannot be rationalized as an adaptive trait by the leading evolutionary theories of human cooperation (in other words, kin selection, reciprocal altruism, indirect reciprocity, and costly signaling theory). However, multilevel selection theories of cultural evolution are consistent with strong reciprocity.     

Three-person game facilitates indirect reciprocity under image scoring

Reputation building plays an important role in the evolution of reciprocal altruism when the same individuals do not interact repeatedly because, by referring to reputation, a reciprocator can know which partners are cooperative and can reciprocate with a cooperator. This reciprocity based on reputation is called indirect reciprocity. Previous studies of indirect reciprocity have focused only on two-person games in which only two individuals participate in a single interaction, and have claimed that indirectly reciprocal cooperation cannot be established under image scoring reputation criterion where the reputation of an individual who has cooperated (defected) becomes good (bad). In this study, we specifically examine three-person games, and reveal that indirectly reciprocal cooperation can be formed and maintained stably, even under image scoring, by a nucleus shield mechanism. In the nucleus shield, reciprocators are a shield that keeps out unconditional defectors, whereas unconditional cooperators are the backbone of cooperation that retains a good reputation among the population.     

Chaos, oscillation and the evolution of indirect reciprocity in n-person games

Evolution of cooperation among genetically unrelated individuals has been of considerable concern in various fields such as biology, economics, and psychology. The evolution of cooperation is often explained by reciprocity. Under reciprocity, cooperation can prevail in a society because a donor of cooperation receives reciprocation from the recipient of the cooperation, called direct reciprocity, or from someone else in the community, called indirect reciprocity. Nowak and Sigmund [1993. Chaos and the evolution of cooperation. Proc. Natl. Acad. Sci. USA 90, 5091-5094] have demonstrated that directly reciprocal cooperation in two-person prisoner's dilemma games with mutation of strategies can be maintained dynamically as periodic or chaotic oscillation. Furthermore, Eriksson and Lindgren [2005. Cooperation driven by mutations in multi-person Prisoner's Dilemma. J. Theor. Biol. 232, 399-409] have reported that directly reciprocal cooperation in n-person prisoner's dilemma games (n>2) can be maintained as periodic oscillation. Is dynamic cooperation observed only in direct reciprocity? Results of this study show that indirectly reciprocal cooperation in n-person prisoner's dilemma games can be maintained dynamically as periodic or chaotic oscillation. This is, to our knowledge, the first demonstration of chaos in indirect reciprocity. Furthermore, the results show that oscillatory dynamics are observed in common in the evolution of reciprocal cooperation whether for direct or indirect.     

Incipient cognition solves the spatial reciprocity conundrum of cooperation

Background

From the simplest living organisms to human societies, cooperation among individuals emerges as a paradox difficult to explain and describe mathematically, although very often observed in reality. Evolutionary game theory offers an excellent toolbar to investigate this issue. Spatial structure has been one of the first mechanisms promoting cooperation; however, alone it only opens a narrow window of viability.

Methodology/Principal Findings

Here we equip individuals with incipient cognitive abilities, and investigate the evolution of cooperation in a spatial world where retaliation, forgiveness, treason and mutualism may coexist, as individuals engage in Prisoner''s Dilemma games. In the model, individuals are able to distinguish their partners and act towards them based on previous interactions. We show how the simplest level of cognition, alone, can lead to the emergence of cooperation.

Conclusions/Significance

Despite the incipient nature of the individuals'' cognitive abilities, cooperation emerges for unprecedented values of the temptation to cheat, being also robust to invasion by cheaters, errors in decision making and inaccuracy of imitation, features akin to many species, including humans.