Tit-for-tat or win-stay, lose-shift?

The repeated Prisoner's Dilemma is usually known as a story of tit-for-tat (TFT). This remarkable strategy has won both of Robert Axelrod's tournaments. TFT does whatever the opponent has done in the previous round. It will cooperate if the opponent has cooperated, and it will defect if the opponent has defected. But TFT has two weaknesses: (i) it cannot correct mistakes (erroneous moves) and (ii) a population of TFT players is undermined by random drift when mutant strategies appear which play always-cooperate (ALLC). Another equally simple strategy called 'win-stay, lose-shift' (WSLS) has neither of these two disadvantages. WSLS repeats the previous move if the resulting payoff has met its aspiration level and changes otherwise. Here, we use a novel approach of stochastic evolutionary game dynamics in finite populations to study mutation-selection dynamics in the presence of erroneous moves. We compare four strategies: always-defect (ALLD), ALLC, TFT and WSLS. There are two possible outcomes: if the benefit of cooperation is below a critical value then ALLD is selected; if the benefit of cooperation is above this critical value then WSLS is selected. TFT is never selected in this evolutionary process, but lowers the selection threshold for WSLS.     

The continuous prisoner's dilemma and the evolution of cooperation through reciprocal altruism with variable investment

Understanding the evolutionary origin and persistence of cooperative behavior is a fundamental biological problem. The standard "prisoner's dilemma," which is the most widely adopted framework for studying the evolution of cooperation through reciprocal altruism between unrelated individuals, does not allow for varying degrees of cooperation. Here we study the continuous iterated prisoner's dilemma, in which cooperative investments can vary continuously in each round. This game has been previously considered for a class of reactive strategies in which current investments are based on the partner's previous investment. In the standard iterated prisoner's dilemma, such strategies are inferior to strategies that take into account both players' previous moves, as is exemplified by the evolutionary dominance of "Pavlov" over "tit for tat." Consequently, we extend the analysis of the continuous prisoner's dilemma to a class of strategies in which current investments depend on previous payoffs and, hence, on both players' previous investments. We show, both analytically and by simulation, that payoff-based strategies, which embody the intuitively appealing idea that individuals invest more in cooperative interactions when they profit from these interactions, provide a natural explanation for the gradual evolution of cooperation from an initially noncooperative state and for the maintenance of cooperation thereafter.     

Reinforcement Learning Explains Conditional Cooperation and Its Moody Cousin

Direct reciprocity, or repeated interaction, is a main mechanism to sustain cooperation under social dilemmas involving two individuals. For larger groups and networks, which are probably more relevant to understanding and engineering our society, experiments employing repeated multiplayer social dilemma games have suggested that humans often show conditional cooperation behavior and its moody variant. Mechanisms underlying these behaviors largely remain unclear. Here we provide a proximate account for this behavior by showing that individuals adopting a type of reinforcement learning, called aspiration learning, phenomenologically behave as conditional cooperator. By definition, individuals are satisfied if and only if the obtained payoff is larger than a fixed aspiration level. They reinforce actions that have resulted in satisfactory outcomes and anti-reinforce those yielding unsatisfactory outcomes. The results obtained in the present study are general in that they explain extant experimental results obtained for both so-called moody and non-moody conditional cooperation, prisoner’s dilemma and public goods games, and well-mixed groups and networks. Different from the previous theory, individuals are assumed to have no access to information about what other individuals are doing such that they cannot explicitly use conditional cooperation rules. In this sense, myopic aspiration learning in which the unconditional propensity of cooperation is modulated in every discrete time step explains conditional behavior of humans. Aspiration learners showing (moody) conditional cooperation obeyed a noisy GRIM-like strategy. This is different from the Pavlov, a reinforcement learning strategy promoting mutual cooperation in two-player situations.     

Stability of tk-/- mutants of L5178Y mouse lymphoma cells in Fischer's medium

The phenotypic stability of over 2000 large- and small-colony trifluorothymidine-resistant (TFTres) variants of L5178Y/tk(+/-)-3.7.2C cells has been examined. All except 4 of 488 spontaneously arising small-colony variants analyzed (0.8%) retained the TFTres phenotype when rechallenged with TFT after growth for several generations in its absence. All of 558 spontaneous large-colony variants, and 440 small-colony or 487 large-colony variants arising from 13 different mutagens showed similar stability. These results attest to the completeness of TFT selection in the mouse-lymphoma assay when used at 1 microgram/ml in Fischer's medium supplemented with heat-inactivated serum and, together with previous cytogenetic and molecular studies, justify considering essentially all such TFTres variants as stable mutants. The implications of these results for those versions of the mouse lymphoma assay that fail to optimize the recovery and scoring of small-colony mutants is discussed.     

Simple adaptive strategy wins the prisoner's dilemma

The prisoner's dilemma has become the leading paradigm to explain the evolution of cooperation among selfish individuals. Here, we present an adaptive strategy that implements new mechanisms to process information about past encounters. The history of moves is summarized in an internal state which then determines the subsequent move. This enables the strategy to adjust its decisions to the character of the current opponent and to adapt the most promising strategic behavior. For this reason, we call such strategies Adaptor. Through evolutionary simulations, we demonstrate that the concept of Adaptor leads to strategic patterns that are (a) highly cooperative when playing against kin, (b) stable in a sense that goes far beyond the concept of evolutionary stability, (c) robust to environmental changes, i.e. variations of the parameter values and finally (d) superior in performance to the most prominent strategies in the literature.     

A tale of two defectors: the importance of standing for evolution of indirect reciprocity

Indirect reciprocity occurs when the cooperative behavior between two individuals is contingent on their previous behavior toward others. Previous theoretical analysis indicates that indirect reciprocity can evolve if individuals use an image-scoring strategy. In this paper, we show that, when errors are added, indirect reciprocity cannot be based on an image-scoring strategy. However, if individuals use a standing strategy, then cooperation through indirect reciprocity is evolutionarily stable. These two strategies differ with respect to the information to which they attend. While image-scoring strategies only need attend to the actions of others, standing strategies also require information about intent. We speculate that this difference may shed light on the evolvability of indirect reciprocity. Additionally, we show that systems of indirect reciprocity are highly sensitive to the availability of information. Finally, we present a model which shows that if indirect reciprocity were to evolve, selection should also favor trusting behavior in relations between strangers.     

Natural selection of memory-one strategies for the iterated prisoner's dilemma

In the iterated Prisoner's Dilemma, mutually cooperative behavior can become established through Darwinian natural selection. In simulated interactions of stochastic memory-one strategies for the Iterated Prisoner's Dilemma, Nowak and Sigmund discovered that cooperative agents using a Pavlov (Win-Stay Lose-Switch) type strategy eventually dominate a random population. This emergence follows more directly from a deterministic dynamical system based on differential reproductive success or natural selection. When restricted to an environment of memory-one agents interacting in iterated Prisoner's Dilemma games with a 1% noise level, the Pavlov agent is the only cooperative strategy and one of very few others that cannot be invaded by a similar strategy. Pavlov agents are trusting but no suckers. They will exploit weakness but repent if punished for cheating.     

Whence tit-for-tat?

Summary In theoretical and empirical studies of the evolution of cooperation, the tit-for-tat strategy (i.e. cooperate unless your partner did not cooperate in the previous interaction) is widely considered to be of central importance. Nevertheless, surprisingly little is known about the conditions in which tit-for-tat appears and disappears across generations in a population of interacting individuals. Here, we apply a newly developed classifier-system model (EvA) in addressing this issue when the key features of interactions are caricatured using the iterated prisoner's dilemma game. Our simple representation of behavioural strategies as algorithms composed of two interacting rules allowed us to determine conditions in which tit-for-tat can replace noncooperative strategies and vice versa. Using direct game-theoretic analysis and simulations with the EvA model, we determined that no strategy is evolutionarily stable, but larger population sizes and longer sequences of interactions between individuals can yield transient dominance by tit-for-tat. Genetic drift among behaviourally equivalent strategies is the key mechanism underlying this dominance. Our analysis suggests that tit-for-tat could be important in nature for cognitively simple organisms of limited memory capacity, in strongly kin-selected or group-selected populations, when interaction sequences between individuals are relatively short, in moderate-sized populations of widely interacting individuals and when defectors appear in the population with moderate frequency.     

The continuous Prisoner's dilemma: II. Linear reactive strategies with noise.

We present a general model for the Continuous Prisoner's Dilemma and study the effect of errors. We find that cooperative strategies that can resist invasion by defectors are optimistic (make high initial offers), generous (always offer more cooperation than the partner did in the previous round) and uncompromising (offer full cooperation only if the partner does). A necessary condition for the emergence of cooperation in the continuous Prisoner's Dilemma with noise is b (1-p)>c, where b and c denote, respectively, the benefit and cost of cooperation, while p is the error rate. This relation can be reformulated as an error threshold: cooperation can only emerge if the probability of making a mistake is below a critical value. We note, however, that cooperation in the continuous Prisoner's Dilemma with noise does not seem to be evolutionarily stable: while it is possible to find cooperative strategies that resist invasion by defectors, such cooperators are generally invaded by more cooperative strategies which eventually yield to defectors. Thus, the long-term evolution of the continuous Prisoner's Dilemma is either characterized by unending cycles or by stable polymorphisms of cooperators and defectors.     

Teaching and learning in a probabilistic prisoner's dilemma

The prisoner's dilemma is much studied in social psychology and decision-making because it models many real-world conflicts. In everyday terms, the choice to 'cooperate' (maximize reward for the group) or 'defect' (maximize reward for the individual) is often attributed to altruistic or selfish motives. Alternatively, behavior during a dilemma may be understood as a function of reinforcement and punishment. Human participants played a prisoner's-dilemma-type game (for points exchangeable for money) with a computer that employed either a teaching strategy (a probabilistic version of tit-for-tat), in which the computer reinforced or punished participants' cooperation or defection, or a learning strategy (a probabilistic version of Pavlov), in which the computer's responses were reinforced and punished by participants' cooperation and defection. Participants learned to cooperate against both computer strategies. However, in a second experiment which varied the context of the game, they learned to cooperate only against one or other strategy; participants did not learn to cooperate against tit-for-tat when they believed that they were playing against another person; participants did not learn to cooperate against Pavlov when the computer's cooperation probability was signaled by a spinner. The results are consistent with the notion that people are biased not only to cooperate or defect on individual social choices, but also to employ one or other strategy of interaction in a pattern across social choices.     

Evolution of cooperation with shared costs and benefits

The quest to determine how cooperation evolves can be based on evolutionary game theory, in spite of the fact that evolutionarily stable strategies (ESS) for most non-zero-sum games are not cooperative. We analyse the evolution of cooperation for a family of evolutionary games involving shared costs and benefits with a continuum of strategies from non-cooperation to total cooperation. This cost-benefit game allows the cooperator to share in the benefit of a cooperative act, and the recipient to be burdened with a share of the cooperator's cost. The cost-benefit game encompasses the Prisoner's Dilemma, Snowdrift game and Partial Altruism. The models produce ESS solutions of total cooperation, partial cooperation, non-cooperation and coexistence between cooperation and non-cooperation. Cooperation emerges from an interplay between the nonlinearities in the cost and benefit functions. If benefits increase at a decelerating rate and costs increase at an accelerating rate with the degree of cooperation, then the ESS has an intermediate level of cooperation. The game also exhibits non-ESS points such as unstable minima, convergent-stable minima and unstable maxima. The emergence of cooperative behaviour in this game represents enlightened self-interest, whereas non-cooperative solutions illustrate the Tragedy of the Commons. Games having either a stable maximum or a stable minimum have the property that small changes in the incentive structure (model parameter values) or culture (starting frequencies of strategies) result in correspondingly small changes in the degree of cooperation. Conversely, with unstable maxima or unstable minima, small changes in the incentive structure or culture can result in a switch from non-cooperation to total cooperation (and vice versa). These solutions identify when human or animal societies have the potential for cooperation and whether cooperation is robust or fragile.     

Joint evolution of altruistic cooperation and dispersal in a metapopulation of small local populations

We investigate the joint evolution of public goods cooperation and dispersal in a metapopulation model with small local populations. Altruistic cooperation can evolve due to assortment and kin selection, and dispersal can evolve because of demographic stochasticity, catastrophes and kin selection. Metapopulation structures resulting in assortment have been shown to make selection for cooperation possible. But how does dispersal affect cooperation and vice versa, when both are allowed to evolve as continuous traits? We found four qualitatively different evolutionary outcomes. (1) Monomorphic evolution to full defection with positive dispersal. (2) Monomorphic evolution to an evolutionarily stable state with positive cooperation and dispersal. In this case, parameter changes selecting for increased cooperation typically also select for increased dispersal. (3) Evolutionary branching can result in the evolutionarily stable coexistence of defectors and cooperators. Although defectors could be expected to disperse more than cooperators, here we show that the opposite case is also possible: Defectors tend to disperse less than cooperators when the total amount of cooperation in the dimorphic population is low enough. (4) Selection for too low cooperation can cause the extinction of the evolving population. For moderate catastrophe rates dispersal needs to be initially very frequent for evolutionary suicide to occur. Although selection for less dispersal in principle could prevent such evolutionary suicide, in most cases this rescuing effect is not sufficient, because selection in the cooperation trait is typically much stronger. If the catastrophe rate is large enough, a part of the boundary of viability can be evolutionarily attracting with respect to both strategy components, in which case evolutionary suicide is expected from all initial conditions.     

Consequences of fluctuating group size for the evolution of cooperation

Studies of cooperation have traditionally focused on discrete games such as the well-known prisoner’s dilemma, in which players choose between two pure strategies: cooperation and defection. Increasingly, however, cooperation is being studied in continuous games that feature a continuum of strategies determining the level of cooperative investment. For the continuous snowdrift game, it has been shown that a gradually evolving monomorphic population may undergo evolutionary branching, resulting in the emergence of a defector strategy that coexists with a cooperator strategy. This phenomenon has been dubbed the ‘tragedy of the commune’. Here we study the effects of fluctuating group size on the tragedy of the commune and derive analytical conditions for evolutionary branching. Our results show that the effects of fluctuating group size on evolutionary dynamics critically depend on the structure of payoff functions. For games with additively separable benefits and costs, fluctuations in group size make evolutionary branching less likely, and sufficiently large fluctuations in group size can always turn an evolutionary branching point into a locally evolutionarily stable strategy. For games with multiplicatively separable benefits and costs, fluctuations in group size can either prevent or induce the tragedy of the commune. For games with general interactions between benefits and costs, we derive a general classification scheme based on second derivatives of the payoff function, to elucidate when fluctuations in group size help or hinder cooperation.     

The efficiency of adapting aspiration levels

Win–stay, lose–shift strategies in repeated games are based on an aspiration level. A move is repeated if and only if the outcome, in the previous round, was satisficing in the sense that the pay-off was at least as high as the aspiration level. We investigate the conditions under which adaptive mechanisms acting on the aspiration level (selection, for instance, or learning) can lead to an efficient outcome; in other words, when can satisficing become optimizing? Analytical results for 2 times 2-games are presented. They suggest that in a large variety of social interactions, self-centred rules (based uniquely on one''s own pay-off) cannot suffice.     

Know when to walk away: contingent movement and the evolution of cooperation

Models of the evolution of cooperation suggest that an important characteristic of successful strategies is the ability to respond contingently to the social environment. A number of mechanisms by which this can be accomplished have been suggested, some of which require relatively complex information processing systems. This research explores relaxing the requirements on information processing while preserving the evolvability of a cooperative strategy. The agent-based computer simulations reported here show that 'Walk Away,' a behavioral rule of extremely limited complexity (move after partner defects), can outperform more complex strategies under a number of conditions. Previous simulations of exit strategies have not examined the effect of implicit and explicit movement costs, different error rates, or the simultaneous presence of TFT and PAVLOV. The simulations reported here establish that the Walk Away strategy resists invasion and can invade a population of defectors at a lower initial frequency than any other strategy. The Walk Away strategy was successful, despite its simplicity, because it exploited aspects of the physical and social environment.     

Reciprocity and communication of partner quality

In a cooperative exchange, the size of a partner''s contribution is likely to depend both on the partner''s ability to supply help and on the partner''s need for help in return. Referring to such needs and abilities as aspects of partner quality, it follows that variation in the amount of help offered in a relationship could transmit information about partner quality. A plausible behaviour might then be to vary the investment in a partner according to available information about partner quality and to invest little in a partner who offers little in return. Thus, regulation of a relationship through communication of partner quality would tend to follow the principle of reciprocity. In an analysis of an iterated game where players have private information about their needs and abilities, I verify this possibility by describing an evolutionarily stable state space strategy, referred to as ''state-dependent reciprocity'', entailing communication of partner quality. Although the evolution of cooperation has been studied in great detail, there has been no previous analysis of communication of needs and abilities in a relationship. It may well be that such communication is of major importance for the evolution of cooperative behaviour in nature.     

Evolution of cooperation driven by reputation-based migration

How cooperation emerges and is stabilized has been a puzzling problem to biologists and sociologists since Darwin. One of the possible answers to this problem lies in the mobility patterns. These mobility patterns in previous works are either random-like or driven by payoff-related properties such as fitness, aspiration, or expectation. Here we address another force which drives us to move from place to place: reputation. To this end, we propose a reputation-based model to explore the effect of migration on cooperation in the contest of the prisoner's dilemma. In this model, individuals earn their reputation scores through previous cooperative behaviors. An individual tends to migrate to a new place if he has a neighborhood of low reputation. We show that cooperation is promoted for relatively large population density and not very large temptation to defect. A higher mobility sensitivity to reputation is always better for cooperation. A longer reputation memory favors cooperation, provided that the corresponding mobility sensitivity to reputation is strong enough. The microscopic perception of the effect of this mechanism is also given. Our results may shed some light on the role played by migration in the emergence and persistence of cooperation.     

Learning and innovative elements of strategy adoption rules expand cooperative network topologies

Cooperation plays a key role in the evolution of complex systems. However, the level of cooperation extensively varies with the topology of agent networks in the widely used models of repeated games. Here we show that cooperation remains rather stable by applying the reinforcement learning strategy adoption rule, Q-learning on a variety of random, regular, small-word, scale-free and modular network models in repeated, multi-agent Prisoner's Dilemma and Hawk-Dove games. Furthermore, we found that using the above model systems other long-term learning strategy adoption rules also promote cooperation, while introducing a low level of noise (as a model of innovation) to the strategy adoption rules makes the level of cooperation less dependent on the actual network topology. Our results demonstrate that long-term learning and random elements in the strategy adoption rules, when acting together, extend the range of network topologies enabling the development of cooperation at a wider range of costs and temptations. These results suggest that a balanced duo of learning and innovation may help to preserve cooperation during the re-organization of real-world networks, and may play a prominent role in the evolution of self-organizing, complex systems.     

Ecological constraints, life history traits and the evolution of cooperative breeding

The ecological constraints hypothesis is widely accepted as an explanation for the evolution of delayed dispersal in cooperatively breeding birds. Intraspecific studies offer the strongest support. Observational studies have demonstrated a positive association between the severity of ecological constraints and the prevalence of cooperation, and experimental studies in which constraints on independent breeding were relaxed resulted in helpers moving to adopt the vacant breeding opportunities. However, this hypothesis has proved less successful in explaining why cooperative breeding has evolved in some species or lineages but not in others. Comparative studies have failed to identify ecological factors that differ consistently between cooperative and noncooperative species. The life history hypothesis, which emphasizes the role of life history traits in the evolution of cooperative breeding, offers a solution to this difficulty. A recent analysis showed that low adult mortality and low dispersal predisposed certain lineages to show cooperative behaviour, given the right ecological conditions. This represents an important advance, not least by offering an explanation for the patchy phylogenetic distribution of cooperative breeding. We discuss the complementary nature of these two hypotheses and suggest that rather than regarding life history traits as predisposing and ecological factors as facilitating cooperation, they are more likely to act in concert. While acknowledging that different cooperative systems may be a consequence of different selective pressures, we suggest that to identify the key differences between cooperative and noncooperative species, a broad constraints hypothesis that incorporates ecological and life history traits in a single measure of 'turnover of breeding opportunities' may provide the most promising avenue for future comparative studies. Copyright 2000 The Association for the Study of Animal Behaviour.     

Evolution of cooperation by reciprocation within structured demes

The iterative two-person Prisoners’ Dilemma game has been generalised to theN-person case. The evolution of cooperation is explored by matching the Tit For Tat (TFT) strategy (Axelrod and Hamilton 1981) against the selfish strategy. Extension of TFT toN-person situations yields a graded set of strategies from the softest TFT, which continues cooperation even if only one of the opponents reciprocates it, to the hardest, which would do so only when all the remaining opponents cooperate. The hardest TFT can go to fixation against the selfish strategy provided it crosses a threshold frequencypc. All the other TFT are invadable by the selfish (D) or the pure defector strategy, while none can invadeD. Yet, provided a thresholdpc is crossed, they can coexist stably withD. AsN, the size of the group increases, the threshold p_c also increases, indicating that the evolution of cooperation is more difficult for larger groups. Under certain conditions, only the soft TFT can coexist stably against the selfish strategyD, while the harder ones cannot. An interesting possibility of a complete takeover of the selfish population by successive invasions by harder and harder TFT strategies is also presented.