Optimality under noise: higher memory strategies for the alternating prisoner's dilemma.

The Alternating Prisoner's Dilemma is a variant of the iterated Prisoner's Dilemma in which the players alternate in the roles of actor and recipient. We searched for strategies which are "optimal" in the Alternating Prisoner's Dilemma with noise (a non-zero probability that a player's decision will be transmitted incorrectly). In order to achieve success against a variety of other strategies, a strategy must be "self-cooperating" (able to achieve mutual cooperation with its clone), "C-exploiting" (able to exploit unconditional cooperators), and "D-unexploitable" (able to resist exploitation by defectors). It must also have high evolutionary "dominance", a general measure of evolutionary performance which considers both resistance to invasion and the ability to invade other strategies. A strategy which meets these optimality criteria can evolve cooperation by invading a population of defectors and establishing a stable cooperative society. Most of the strategies commonly discussed in the Alternating Prisoner's Dilemma literature are low-memory strategies such as Tit For Tat, Pavlov, and Firm But Fair, but none of these strategies can simultaneously meet all of the optimality criteria. However, we discovered a class of higher memory "Firm Pavlov" strategies, which not only meet our stringent optimality criteria, but also achieve remarkable success in round-robin tournaments and evolutionary interactions. These higher memory strategies are friendly enough to cooperate with their clone, pragmatic enough to exploit unconditional cooperators, and wary enough to resist exploitation by defectors: they are truly "optimal under noise" in the Alternating Prisoner's Dilemma.     

The continuous prisoner's dilemma and the evolution of cooperation through reciprocal altruism with variable investment

Understanding the evolutionary origin and persistence of cooperative behavior is a fundamental biological problem. The standard "prisoner's dilemma," which is the most widely adopted framework for studying the evolution of cooperation through reciprocal altruism between unrelated individuals, does not allow for varying degrees of cooperation. Here we study the continuous iterated prisoner's dilemma, in which cooperative investments can vary continuously in each round. This game has been previously considered for a class of reactive strategies in which current investments are based on the partner's previous investment. In the standard iterated prisoner's dilemma, such strategies are inferior to strategies that take into account both players' previous moves, as is exemplified by the evolutionary dominance of "Pavlov" over "tit for tat." Consequently, we extend the analysis of the continuous prisoner's dilemma to a class of strategies in which current investments depend on previous payoffs and, hence, on both players' previous investments. We show, both analytically and by simulation, that payoff-based strategies, which embody the intuitively appealing idea that individuals invest more in cooperative interactions when they profit from these interactions, provide a natural explanation for the gradual evolution of cooperation from an initially noncooperative state and for the maintenance of cooperation thereafter.     

Asymmetric interaction will facilitate the evolution of cooperation

Explaining the evolution of cooperation remains one of the greatest problems for both biology and social science. The classical theories of cooperation suggest that cooperation equilibrium or evolutionary stable strategy between partners can be maintained through genetic similarity or reciprocity relatedness. These classical theories are based on an assumption that partners interact symmetrically with equal payoffs in a game of cooperation interaction. However, the payoff between partners is usually not equal and therefore they often interact asymmetrically in real cooperative systems. With the Hawk-Dove model, we find that the probability of cooperation between cooperative partners will depend closely on the payoff ratio. The higher the payoff ratio between recipients and cooperative actors, the greater will be the probability of cooperation interaction between involved partners. The greatest probability of conflict between cooperative partners will occur when the payoff between partners is equal. The results show that this asymmetric relationship is one of the key dynamics of the evolution of cooperation, and that pure cooperation strategy (i.e., Nash equilibrium) does not exist in asymmetrical cooperation systems, which well explains the direct conflict observed in almost all of the well documented cooperation systems. The model developed here shows that the cost-to-benefit ratio of cooperation is also negatively correlated with the probability of cooperation interaction. A smaller cost-to-benefit ratio of cooperation might be created by the limited dispersal ability or exit cost of the partners involved, and it will make the punishment of the non-cooperative individuals by the recipient more credible, and therefore make it more possible to maintain stable cooperation interaction.     

Evolutionary dynamics of the continuous iterated prisoner's dilemma

The iterated prisoner's dilemma (IPD) has been widely used in the biological and social sciences to model dyadic cooperation. While most of this work has focused on the discrete prisoner's dilemma, in which actors choose between cooperation and defection, there has been some analysis of the continuous IPD, in which actors can choose any level of cooperation from zero to one. Here, we analyse a model of the continuous IPD with a limited strategy set, and show that a generous strategy achieves the maximum possible payoff against its own type. While this strategy is stable in a neighborhood of the equilibrium point, the equilibrium point itself is always vulnerable to invasion by uncooperative strategies, and hence subject to eventual destabilization. The presence of noise or errors has no effect on this result. Instead, generosity is favored because of its role in increasing contributions to the most efficient level, rather than in counteracting the corrosiveness of noise. Computer simulation using a single-locus infinite alleles Gaussian mutation model suggest that outcomes ranging from a stable cooperative polymorphism to complete collapse of cooperation are possible depending on the magnitude of the mutational variance. Also, making the cost of helping a convex function of the amount of help provided makes it more difficult for cooperative strategies to invade a non-cooperative equilibrium, and for the cooperative equilibrium to resist destabilization by non-cooperative strategies. Finally, we demonstrate that a much greater degree of assortment is required to destabilize a non-cooperative equilibrium in the continuous IPD than in the discrete IPD. The continuous model outlined here suggests that incremental amounts of cooperation lead to rapid decay of cooperation and thus even a large degree of assortment will not be sufficient to allow cooperation to increase when cooperators are rare. The extreme degree of assortment required to destabilize the non-cooperative equilibrium, as well as the instability of the cooperative equilibrium, may help explain why cooperation in Prisoner's Dilemmas is so rare in nature.     

Friendship, cliquishness, and the emergence of cooperation

The evolution of cooperation is a central problem in biology and the social sciences. While theoretical work using the iterated prisoner's dilemma (IPD) has shown that cooperation among non-kin can be sustained among reciprocal strategies (i.e. tit-for-tat), these results are sensitive to errors in strategy execution, cyclical invasions by free riders, and the specific ecology of strategies. Moreover, the IPD assumes that a strategy's probability of playing the PD game with other individuals is independent of the decisions made by others. Here, we remove the assumption of independent pairing by studying a more plausible cooperative dilemma in which players can preferentially interact with a limited set of known partners and also deploy longer-term accounting strategies that can counteract the effects of random errors. We show that cooperative strategies readily emerge and persist in a range of noisy environments, with successful cooperative strategies (henceforth, cliquers) maintaining medium-term memories for partners and low thresholds for acceptable cooperation (i.e. forgiveness). The success of these strategies relies on their cliquishness-a propensity to defect with strangers if they already have an adequate number of partners. Notably, this combination of medium-term accounting, forgiveness, and cliquishness fits with empirical studies of friendship and other long-term relationships among humans.     

A tale of two defectors: the importance of standing for evolution of indirect reciprocity

Indirect reciprocity occurs when the cooperative behavior between two individuals is contingent on their previous behavior toward others. Previous theoretical analysis indicates that indirect reciprocity can evolve if individuals use an image-scoring strategy. In this paper, we show that, when errors are added, indirect reciprocity cannot be based on an image-scoring strategy. However, if individuals use a standing strategy, then cooperation through indirect reciprocity is evolutionarily stable. These two strategies differ with respect to the information to which they attend. While image-scoring strategies only need attend to the actions of others, standing strategies also require information about intent. We speculate that this difference may shed light on the evolvability of indirect reciprocity. Additionally, we show that systems of indirect reciprocity are highly sensitive to the availability of information. Finally, we present a model which shows that if indirect reciprocity were to evolve, selection should also favor trusting behavior in relations between strangers.     

The stability of cooperation involving variable investment

In this paper, we attempt to reconcile the results of several studies which have investigated the evolution of cooperation between non-relatives in systems where investment in partners can vary. In contrast to previous proposals, we show for the first time that variable-investment cooperation can be readily maintained in inter-species mutualistic relationships even in the absence of spatial structure, but that the stability of this interaction is dependent on the particular investment-response rule that is employed. By allowing the evolution of investment-response parameters in both inter- and intra-specific versions of the continuous variable-investment Prisoners' Dilemma we show that, in the absence of further factors, the raise-the stakes (RTS) strategy is likely to evolve into a simpler, variable investment form of Give-as-Good-as-you-Get that initially offers a high fixed amount and subsequently matches its partner's investment. Nevertheless, we demonstrate that genuine RTS-like strategies will still be selected if two intuitively reasonable conditions hold: if individuals are limited in terms of the total they can invest in cooperative actions over their lifetimes, and if there are always some individuals in a population that cannot cooperate.     

Evolutionary game theory meets social science: Is there a unifying rule for human cooperation?

Evolutionary game theory has shown that human cooperation thrives in different types of social interactions with a PD structure. Models treat the cooperative strategies within the different frameworks as discrete entities and sometimes even as contenders. Whereas strong reciprocity was acclaimed as superior to classic reciprocity for its ability to defeat defectors in public goods games, recent experiments and simulations show that costly punishment fails to promote cooperation in the IR and DR games, where classic reciprocity succeeds. My aim is to show that cooperative strategies across frameworks are capable of a unified treatment, for they are governed by a common underlying rule or norm. An analysis of the reputation and action rules that govern some representative cooperative strategies both in models and in economic experiments confirms that the different frameworks share a conditional action rule and several reputation rules. The common conditional rule contains an option between costly punishment and withholding benefits that provides alternative enforcement methods against defectors. Depending on the framework, individuals can switch to the appropriate strategy and method of enforcement. The stability of human cooperation looks more promising if one mechanism controls successful strategies across frameworks.     

The continuous prisoner's dilemma: I. linear reactive strategies.

We present a general model for the Prisoner's Dilemma in which variable degrees of cooperation are possible, and payoffs are scaled accordingly. We describe a continuous strategy space, and divide this space into strategy families. We derive the payoff function for these families analytically, and study the evolutionary outcome when a wide range of strategies play against each other. Our results show that the initial degree of cooperation offered by a strategy is a decisive factor for evolutionary robustness: the most successful strategies in our model offer full cooperation as an initial move, but thereafter cooperate fully only if their opponent does the same. These strategies gradually raise the stakes when playing a strategy which is initially reticent to cooperate, but differ from the strategies predicted by other continuous models in that they are not only generous, but are also consistently optimistic and uncompromising.     

Uncertainty causes humans to use social heuristics and to cooperate more: An experiment among Belgian university students

When humans engage in social interactions, they are often uncertain about what the possible outcomes are. Because of this, highly sophisticated cooperation strategies may not be very effective. Indeed, some models instead predict the emergence of ‘social heuristics’: simple cooperation strategies that perform well across a range of different situations. Here, we put these predictions to the test in a large-scale interactive decision making experiment. We confronted participants (mostly Belgian university students) with a broad range of cooperative interactions, systematically varying the uncertainty participants had about the consequences of cooperating. As expected, we find that uncertainty about the payoff consequences of cooperation causes individuals to use social heuristics. Additionally, these heuristics directly cause a marked increase in cooperation compared to the treatment without uncertainty, even in situations where cooperation can never be beneficial. These findings provide a new explanation for why human social behavior often violates the standard predictions of economic and evolutionary theory.     

Cooperative personalities and social niche specialization in female meerkats

The social niche specialization hypothesis predicts that group‐living animals should specialize in particular social roles to avoid social conflict, resulting in alternative life‐history strategies for different roles. Social niche specialization, coupled with role‐specific life‐history trade‐offs, should thus generate between‐individual differences in behaviour that persist through time, or distinct personalities, as individuals specialize in particular nonoverlapping social roles. We tested for support for the social niche specialization hypothesis in cooperative personality traits in wild female meerkats (Suricata suricatta) that compete for access to dominant social roles. As cooperation is costly and dominance is acquired by heavier females, we predicted that females that ultimately acquired dominant roles would show noncooperative personality types early in life and before and after role acquisition. Although we found large individual differences in repeatable cooperative behaviours, there was no indication that individuals that ultimately acquired dominance differed from unsuccessful individuals in their cooperative behaviour. Early‐life behaviour did not predict social role acquisition later in life, nor was cooperative behaviour before and after role acquisition correlated in the same individuals. We suggest that female meerkats do not show social niche specialization resulting in cooperative personalities, but that they exhibit an adaptive response in personality at role acquisition.     

On the further integration of cooperative breeding and cooperation theory

We present a synopsis about the commentaries to the target article "Integrating cooperative breeding into theoretical concepts of cooperation", in which we attempted to integrate general mechanisms to explain cooperative behaviour among unrelated individuals with classic concepts to explain helping behaviour in cooperative breeders that do not invoke kin-based benefits. Here we (1) summarize the positions of the commentators concerning the main issues we raised in the target article and discuss important criticisms and extensions. (2) We relate our target article to some recent reviews on the evolution of cooperation and, (3) clarify how we use terminology with regard to cooperation and cooperative behaviour. (4) We discuss several aspects that were raised with respect to cooperative interactions including by-product mutualism, generalised reciprocity and multi-level selection and, (5) examine the alternatives to our classification scheme as proposed by some commentaries. (6) Finally, we highlight several aspects that might hinder the application of game theoretical mechanisms of cooperation in cooperatively breeding systems. Although there is broad agreement that cooperative breeding theory should be integrated within the more general concepts of cooperation, there is some debate about how this may be achieved. We conclude that the contributions in this special issue provide a fruitful first step and ample suggestions for future directions with regard to a more unified framework of cooperation in cooperative breeders.     

Resolving the iterated prisoner's dilemma: theory and reality

Pairs of unrelated individuals face a prisoner's dilemma if cooperation is the best mutual outcome, but each player does best to defect regardless of his partner's behaviour. Although mutual defection is the only evolutionarily stable strategy in one-shot games, cooperative solutions based on reciprocity can emerge in iterated games. Among the most prominent theoretical solutions are the so-called bookkeeping strategies, such as tit-for-tat, where individuals copy their partner's behaviour in the previous round. However, the lack of empirical data conforming to predicted strategies has prompted the suggestion that the iterated prisoner's dilemma (IPD) is neither a useful nor realistic basis for investigating cooperation. Here, we discuss several recent studies where authors have used the IPD framework to interpret their data. We evaluate the validity of their approach and highlight the diversity of proposed solutions. Strategies based on precise accounting are relatively uncommon, perhaps because the full set of assumptions of the IPD model are rarely satisfied. Instead, animals use a diverse array of strategies that apparently promote cooperation, despite the temptation to cheat. These include both positive and negative reciprocity, as well as long-term mutual investments based on 'friendships'. Although there are various gaps in these studies that remain to be filled, we argue that in most cases, individuals could theoretically benefit from cheating and that cooperation cannot therefore be explained with the concept of positive pseudo-reciprocity. We suggest that by incorporating empirical data into the theoretical framework, we may gain fundamental new insights into the evolution of mutual reciprocal investment in nature.     

Different reactions to adverse neighborhoods in games of cooperation

In social dilemmas, cooperation among randomly interacting individuals is often difficult to achieve. The situation changes if interactions take place in a network where the network structure jointly evolves with the behavioral strategies of the interacting individuals. In particular, cooperation can be stabilized if individuals tend to cut interaction links when facing adverse neighborhoods. Here we consider two different types of reaction to adverse neighborhoods, and all possible mixtures between these reactions. When faced with a gloomy outlook, players can either choose to cut and rewire some of their links to other individuals, or they can migrate to another location and establish new links in the new local neighborhood. We find that in general local rewiring is more favorable for the evolution of cooperation than emigration from adverse neighborhoods. Rewiring helps to maintain the diversity in the degree distribution of players and favors the spontaneous emergence of cooperative clusters. Both properties are known to favor the evolution of cooperation on networks. Interestingly, a mixture of migration and rewiring is even more favorable for the evolution of cooperation than rewiring on its own. While most models only consider a single type of reaction to adverse neighborhoods, the coexistence of several such reactions may actually be an optimal setting for the evolution of cooperation.     

Cooperation prevails when individuals adjust their social ties

Conventional evolutionary game theory predicts that natural selection favours the selfish and strong even though cooperative interactions thrive at all levels of organization in living systems. Recent investigations demonstrated that a limiting factor for the evolution of cooperative interactions is the way in which they are organized, cooperators becoming evolutionarily competitive whenever individuals are constrained to interact with few others along the edges of networks with low average connectivity. Despite this insight, the conundrum of cooperation remains since recent empirical data shows that real networks exhibit typically high average connectivity and associated single-to-broad–scale heterogeneity. Here, a computational model is constructed in which individuals are able to self-organize both their strategy and their social ties throughout evolution, based exclusively on their self-interest. We show that the entangled evolution of individual strategy and network structure constitutes a key mechanism for the sustainability of cooperation in social networks. For a given average connectivity of the population, there is a critical value for the ratio W between the time scales associated with the evolution of strategy and of structure above which cooperators wipe out defectors. Moreover, the emerging social networks exhibit an overall heterogeneity that accounts very well for the diversity of patterns recently found in acquired data on social networks. Finally, heterogeneity is found to become maximal when W reaches its critical value. These results show that simple topological dynamics reflecting the individual capacity for self-organization of social ties can produce realistic networks of high average connectivity with associated single-to-broad–scale heterogeneity. On the other hand, they show that cooperation cannot evolve as a result of “social viscosity” alone in heterogeneous networks with high average connectivity, requiring the additional mechanism of topological co-evolution to ensure the survival of cooperative behaviour.     

When cooperation begets cooperation: the role of key individuals in galvanizing support

Life abounds with examples of conspecifics actively cooperating to a common end, despite conflicts of interest being expected concerning how much each individual should contribute. Mathematical models typically find that such conflict can be resolved by partial-response strategies, leading investors to contribute relatively equitably. Using a case study approach, we show that such model expectations can be contradicted in at least four disparate contexts: (i) bi-parental care; (ii) cooperative breeding; (iii) cooperative hunting; and (iv) human cooperation. We highlight that: (a) marked variation in contributions is commonplace; and (b) individuals can often respond positively rather than negatively to the contributions of others. Existing models have surprisingly limited power in explaining these phenomena. Here, we propose that, although among-individual variation in cooperative contributions will be influenced by differential costs and benefits, there is likely to be a strong genetic or epigenetic component. We then suggest that selection can maintain high investors (key individuals) when their contributions promote support by increasing the benefits and/or reducing the costs for others. Our intentions are to raise awareness in—and provide testable hypotheses of—two of the most poorly understood, yet integral, questions regarding cooperative ventures: why do individuals vary in their contributions and when does cooperation beget cooperation?     

Iterated prisoner's dilemma in an asocial world dominated by loners, not by defectors

Cooperation among genetically unrelated individuals can arise when pairs of individuals interact repeatedly in the Prisoner’s Dilemma. However, the conditions allowing the evolution of reciprocal cooperation become extremely restrictive as the size of the cooperative group increases, because defectors can exploit cooperators more efficiently in larger groups. Here we consider three strategies: Tit for Tat, defector, and loner. Loner beats defector in a non-cooperative world. However, a cooperative strategy Tit for Tat (TFT₀) that stops cooperation after the first iteration when there is at least one defector in the group, can invade a world of loners, even in sizable groups, if both the TFT₀ and the defector strategies arise at the same frequency by mutation.     

Contagion of Cooperation in Static and Fluid Social Networks

Cooperation is essential for successful human societies. Thus, understanding how cooperative and selfish behaviors spread from person to person is a topic of theoretical and practical importance. Previous laboratory experiments provide clear evidence of social contagion in the domain of cooperation, both in fixed networks and in randomly shuffled networks, but leave open the possibility of asymmetries in the spread of cooperative and selfish behaviors. Additionally, many real human interaction structures are dynamic: we often have control over whom we interact with. Dynamic networks may differ importantly in the goals and strategic considerations they promote, and thus the question of how cooperative and selfish behaviors spread in dynamic networks remains open. Here, we address these questions with data from a social dilemma laboratory experiment. We measure the contagion of both cooperative and selfish behavior over time across three different network structures that vary in the extent to which they afford individuals control over their network ties. We find that in relatively fixed networks, both cooperative and selfish behaviors are contagious. In contrast, in more dynamic networks, selfish behavior is contagious, but cooperative behavior is not: subjects are fairly likely to switch to cooperation regardless of the behavior of their neighbors. We hypothesize that this insensitivity to the behavior of neighbors in dynamic networks is the result of subjects’ desire to attract new cooperative partners: even if many of one’s current neighbors are defectors, it may still make sense to switch to cooperation. We further hypothesize that selfishness remains contagious in dynamic networks because of the well-documented willingness of cooperators to retaliate against selfishness, even when doing so is costly. These results shed light on the contagion of cooperative behavior in fixed and fluid networks, and have implications for influence-based interventions aiming at increasing cooperative behavior.     

Critical dynamics in the evolution of stochastic strategies for the iterated prisoner's dilemma

The observed cooperation on the level of genes, cells, tissues, and individuals has been the object of intense study by evolutionary biologists, mainly because cooperation often flourishes in biological systems in apparent contradiction to the selfish goal of survival inherent in Darwinian evolution. In order to resolve this paradox, evolutionary game theory has focused on the Prisoner's Dilemma (PD), which incorporates the essence of this conflict. Here, we encode strategies for the iterated Prisoner's Dilemma (IPD) in terms of conditional probabilities that represent the response of decision pathways given previous plays. We find that if these stochastic strategies are encoded as genes that undergo Darwinian evolution, the environmental conditions that the strategies are adapting to determine the fixed point of the evolutionary trajectory, which could be either cooperation or defection. A transition between cooperative and defective attractors occurs as a function of different parameters such as mutation rate, replacement rate, and memory, all of which affect a player's ability to predict an opponent's behavior. These results imply that in populations of players that can use previous decisions to plan future ones, cooperation depends critically on whether the players can rely on facing the same strategies that they have adapted to. Defection, on the other hand, is the optimal adaptive response in environments that change so quickly that the information gathered from previous plays cannot usefully be integrated for a response.     

A simple learning strategy that realizes robust cooperation better than Pavlov in Iterated Prisoners' Dilemma

Pavlov was proposed as a leading strategy for realizing cooperation because it dominates over a long period in evolutionary computer simulations of the Iterated Prisoners' Dilemma. However, our numerical calculations reveal that Pavlov and also any other cooperative strategy are not evolutionarily stable among all stochastic strategies with memory of only one previous move. We propose simple learning based on reinforcement. The learner changes its internal state, depending on an evaluation of whether the score in the previous round is larger than a critical value (aspiration level), which is genetically fixed. The current internal state decides the learner's move, but we found that the aspiration level determines its final behavior. The cooperative variant, having an intermediate aspiration level, is not an evolutionarily stable strategy (ESS) when evaluation is binary (good or bad). However, when the evaluation is quantified some cooperative variants can invade not only All-C, Tit-For-Tat (TFT), and Pavlov but also noncooperative variants with different aspiration levels. Moreover, they establish robust cooperation, which is evolutionarily stable against invasion by All-C, All-D, TFT, Pavlov, and noncooperative variants, and they receive a high score even when the error rate is high. Our results suggest that mutual cooperation can be maintained when players have a primitive learning ability. Received: June 23, 2000 / Accepted: October 1, 2000