A Theoretical Analysis of Temporal Difference Learning in the Iterated Prisoner’s Dilemma Game

Direct reciprocity is a chief mechanism of mutual cooperation in social dilemma. Agents cooperate if future interactions with the same opponents are highly likely. Direct reciprocity has been explored mostly by evolutionary game theory based on natural selection. Our daily experience tells, however, that real social agents including humans learn to cooperate based on experience. In this paper, we analyze a reinforcement learning model called temporal difference learning and study its performance in the iterated Prisoner’s Dilemma game. Temporal difference learning is unique among a variety of learning models in that it inherently aims at increasing future payoffs, not immediate ones. It also has a neural basis. We analytically and numerically show that learners with only two internal states properly learn to cooperate with retaliatory players and to defect against unconditional cooperators and defectors. Four-state learners are more capable of achieving a high payoff against various opponents. Moreover, we numerically show that four-state learners can learn to establish mutual cooperation for sufficiently small learning rates.     

Strong reciprocity, human cooperation, and the enforcement of social norms

This paper provides strong evidence challenging the self-interest assumption that dominates the behavioral sciences and much evolutionary thinking. The evidence indicates that many people have a tendency to voluntarily cooperate, if treated fairly, and to punish noncooperators. We call this behavioral propensity “strong reciprocity” and show empirically that it can lead to almost universal cooperation in circumstances in which purely self-interested behavior would cause a complete breakdown of cooperation. In addition, we show that people are willing to punish those who behaved unfairly towards a third person or who defected in a Prisoner’s Dilemma game with a third person. This suggests that strong reciprocity is a powerful device for the enforcement of social norms involving, for example, food sharing or collective action. Strong reciprocity cannot be rationalized as an adaptive trait by the leading evolutionary theories of human cooperation (in other words, kin selection, reciprocal altruism, indirect reciprocity, and costly signaling theory). However, multilevel selection theories of cultural evolution are consistent with strong reciprocity.     

Analytical Results for Individual and Group Selection of Any Intensity

The idea of evolutionary game theory is to relate the payoff of a game to reproductive success (= fitness). An underlying assumption in most models is that fitness is a linear function of the payoff. For stochastic evolutionary dynamics in finite populations, this leads to analytical results in the limit of weak selection, where the game has a small effect on overall fitness. But this linear function makes the analysis of strong selection difficult. Here, we show that analytical results can be obtained for any intensity of selection, if fitness is defined as an exponential function of payoff. This approach also works for group selection (= multi-level selection). We discuss the difference between our approach and that of inclusive fitness theory.     

Human altruism: economic, neural, and evolutionary perspectives

Human cooperation represents a spectacular outlier in the animal world. Unlike other creatures, humans frequently cooperate with genetically unrelated strangers, often in large groups, with people they will never meet again, and when reputation gains are small or absent. Experimental evidence and evolutionary models suggest that strong reciprocity, the behavioral propensity for altruistic punishment and altruistic rewarding, is of key importance for human cooperation. Here, we review both evidence documenting altruistic punishment and altruistic cooperation and recent brain imaging studies that combine the powerful tools of behavioral game theory with neuroimaging techniques. These studies show that mutual cooperation and the punishment of defectors activate reward related neural circuits, suggesting that evolution has endowed humans with proximate mechanisms that render altruistic behavior psychologically rewarding.     

Strong reciprocity and human sociality

Human groups maintain a high level of sociality despite a low level of relatedness among group members. This paper reviews the evidence for an empirically identifiable form of prosocial behavior in humans, which we call "strong reciprocity", that may in part explain human sociality. A strong reciprocator is predisposed to cooperate with others and punish non-cooperators, even when this behavior cannot be justified in terms of extended kinship or reciprocal altruism. We present a simple model, stylized but plausible, of the evolutionary emergence of strong reciprocity.     

An experimental study of strong reciprocity in bacteria

Strong reciprocity, whereby cooperators punish non-cooperators, may help to explain the evolutionary success of cooperative behaviours. However, theory suggests that selection for strong reciprocity can depend upon tight genetic linkage between cooperation and punishment, to avoid the strategy being outcompeted by non-punishing cooperators. We tested this hypothesis using experimental populations of the bacterium Pseudomonas aeruginosa, which cooperate by producing iron-scavenging siderophores and, in this context, punish non-cooperators with toxins. Consistent with theory, we show that cooperative punishers can indeed invade cheats, but only when the traits are tightly linked. These results emphasize that punishment is only likely to be favoured when the punishment itself leads to a direct or indirect fitness benefit to the actor.     

Cooperation under indirect reciprocity and imitative trust

Indirect reciprocity, a key concept in behavioral experiments and evolutionary game theory, provides a mechanism that allows reciprocal altruism to emerge in a population of self-regarding individuals even when repeated interactions between pairs of actors are unlikely. Recent empirical evidence show that humans typically follow complex assessment strategies involving both reciprocity and social imitation when making cooperative decisions. However, currently, we have no systematic understanding of how imitation, a mechanism that may also generate negative effects via a process of cumulative advantage, affects cooperation when repeated interactions are unlikely or information about a recipient's reputation is unavailable. Here we extend existing evolutionary models, which use an image score for reputation to track how individuals cooperate by contributing resources, by introducing a new imitative-trust score, which tracks whether actors have been the recipients of cooperation in the past. We show that imitative trust can co-exist with indirect reciprocity mechanisms up to a threshold and then cooperation reverses -revealing the elusive nature of cooperation. Moreover, we find that when information about a recipient's reputation is limited, trusting the action of third parties towards her (i.e. imitating) does favor a higher collective cooperation compared to random-trusting and share-alike mechanisms. We believe these results shed new light on the factors favoring social imitation as an adaptive mechanism in populations of cooperating social actors.     

From reciprocity to unconditional altruism through signalling benefits

Cooperation among genetically unrelated individuals is commonly explained by the potential for future reciprocity or by the risk of being punished by group members. However, unconditional altruism is more difficult to explain. We demonstrate that unconditional altruism can evolve as a costly signal of individual quality (i.e. a handicap) as a consequence of reciprocal altruism. This is because the emergent correlation between altruism and individual quality in reciprocity games can facilitate the use of altruism as a quality indicator in a much wider context, outside the reciprocity game, thus affecting its further evolution through signalling benefits. Our model, based on multitype evolutionary game theory shows that, when the additive signalling benefit of donating help exceeds the cost for only some individuals (of high-quality state) but not for others (of low-quality state), the population possesses an evolutionarily stable strategy (ESS) profile wherein high-quality individuals cooperate unconditionally while low-quality individuals defect or play tit-for-tat (TfT). Hence, as predicted by Zahavi's handicap model, signalling benefits of altruistic acts can establish a stable generosity by high-quality individuals that no longer depends on the probability of future reciprocation or punishment.     

Repeated games and direct reciprocity under active linking

Direct reciprocity relies on repeated encounters between the same two individuals. Here we examine the evolution of cooperation under direct reciprocity in dynamically structured populations. Individuals occupy the vertices of a graph, undergoing repeated interactions with their partners via the edges of the graph. Unlike the traditional approach to evolutionary game theory, where individuals meet at random and have no control over the frequency or duration of interactions, we consider a model in which individuals differ in the rate at which they seek new interactions. Moreover, once a link between two individuals has formed, the productivity of this link is evaluated. Links can be broken off at different rates. Whenever the active dynamics of links is sufficiently fast, population structure leads to a simple transformation of the payoff matrix, effectively changing the game under consideration, and hence paving the way for reciprocators to dominate defectors. We derive analytical conditions for evolutionary stability.     

The return of reciprocity: a psychological approach to the evolution of cooperation

Recent developments in evolutionary game theory argue the superiority of punishment over reciprocity as accounts of large-scale human cooperation. I introduce a distinction between a behavioral and a psychological perspective on reciprocity and punishment to question this view. I examine a narrow and a wide version of a psychological mechanism for reciprocity and conclude that a narrow version is clearly distinguishable from punishment, but inadequate for humans; whereas a wide version is applicable to humans but indistinguishable from punishment. The mechanism for reciprocity in humans emerges as a meta-norm that governs both retaliation and punishment. I make predictions open to empirical investigation to confirm or disconfirm this view.

Cooperation and the scale of competition in humans

Explaining cooperation is one of the greatest challenges for evolutionary biology. It is particularly a problem in species such as humans, where there is cooperation between nonrelatives. Numerous possible solutions have been suggested for the problem of cooperation between nonrelatives, including punishment, policing, and various forms of reciprocity. Here, we suggest that local competition for resources can pose a problem for these hypotheses, analogous to how it can select against cooperation between relatives. We extend the prisoner's dilemma (PD) game to show that local competition between interacting individuals can reduce selection for cooperation between nonrelatives. This is because, with local competition, fitness is relative to social partners, and cooperation benefits social partners. We then test whether nonrelated humans adjust their level of cooperation facultatively in response to the scale of competition when playing the PD for cash prizes. As predicted, we found that individuals were less likely to cooperate when competition was relatively local. Cooperation between humans will therefore be most likely when repeated interactions take place on a local scale between small numbers of people, and competition for resources takes place on a more global scale among large numbers of people.     

The development of contingent reciprocity in children

Cooperation between nonrelatives is common in humans. Reciprocal altruism is a plausible evolutionary mechanism for cooperation within unrelated pairs, as selection may favor individuals who selectively cooperate with those who have cooperated with them in the past. Reciprocity is often observed in humans, but there is only limited evidence of reciprocal altruism in other primate species, raising questions about the origins of human reciprocity. Here, we explore how reciprocity develops in a sample of American children ranging from 3 to 7.5 years of age, and also compare children's behavior to that of chimpanzees in prior studies to gain insight into the phylogeny of human reciprocity. Children show a marked tendency to respond contingently to both prosocial and selfish acts, patterns that have not been seen among chimpanzees in prior studies. Our results show that reciprocity increases markedly with age in this population of children, and by about 5.5 years of age children consistently match the previous behavior of their partners.     

MATHEMATICS OF KIN‐ AND GROUP‐SELECTION: FORMALLY EQUIVALENT?

Evolutionary game theory is a general mathematical framework that describes the evolution of social traits. This framework forms the basis of many multilevel selection models and is also frequently used to model evolutionary dynamics on networks. Kin selection, which was initially restricted to describe social interactions between relatives, has also led to a broader mathematical approach, inclusive fitness, that can not only describe social evolution among relatives, but also in group structured populations or on social networks. It turns out that the underlying mathematics of game theory is fundamentally different from the approach of inclusive fitness. Thus, both approaches—evolutionary game theory and inclusive fitness—can be helpful to understand the evolution of social traits in group structured or spatially extended populations.     

A friend in need is a friend indeed: Need-based sharing,rather than cooperative assortment,predicts experimental resource transfers among Agta hunter-gatherers

Despite much theorizing, the evolutionary reasons why humans cooperate extensively with unrelated individuals are still largely unknown. While reciprocity explains many instances of non-kin cooperation, much remains to be understood. A recent suite of models based upon ‘cooperative assortativity’ suggest that non-kin cooperation can evolve if individuals preferentially assort with certain cooperative phenotypes, such as helping those who help others. Here, we test these assortative hypotheses among the Agta, a population of Filipino hunter-gatherers, using an experimental resource allocation game in which individuals divide resources between themselves and camp-mates. Individuals preferentially shared with less cooperative individuals, arguing against cooperative assortativity as a mechanism sustaining resource transfers in this population. Rather, sharing was often based on the recipient's level of need, in addition to kin-based transfers and reciprocal sharing. Contrary to several recent theoretical accounts, in this real-world setting we find no evidence for cooperative assortativity influencing patterns of cooperation. These results may reflect the demands of living in a foraging ecology characterized by high resource stochasticity, necessitating need-based sharing as a system of long-term reciprocity to mitigate repeated subsistence shortfalls.     

Were neandertal and modern human cranial differences produced by natural selection or genetic drift?

Most evolutionary explanations for cranial differences between Neandertals and modern humans emphasize adaptation by natural selection. Features of the crania of Neandertals could be adaptations to the glacial climate of Pleistocene Europe or to the high mechanical strains produced by habitually using the front teeth as tools, while those of modern humans could be adaptations for articulate speech production. A few researchers have proposed non-adaptive explanations. These stress that isolation between Neandertal and modern human populations would have lead to cranial diversification by genetic drift (chance changes in the frequencies of alleles at genetic loci contributing to variation in cranial morphology). Here we use a variety of statistical tests founded on explicit predictions from quantitative- and population-genetic theory to show that genetic drift can explain cranial differences between Neandertals and modern humans. These tests are based on thirty-seven standard cranial measurements from a sample of 2524 modern humans from 30 populations and 20 Neandertal fossils. As a further test, we compare our results for modern human cranial measurements with those for a genetic dataset consisting of 377 microsatellites typed for a sample of 1056 modern humans from 52 populations. We conclude that rather than requiring special adaptive accounts, Neandertal and modern human crania may simply represent two outcomes from a vast space of random evolutionary possibilities.     

Stochastic sampling of interaction partners versus deterministic payoff assignment

Evolutionary game dynamics describes how successful strategies spread in a population. In well-mixed populations, the usual assumption, e.g. underlying the replicator dynamics, is that individuals obtain a payoff from interactions with a representative sample of the population. This determines their fitness. Here, we analyze a situation in which payoffs are obtained through a single interaction, so that individuals of the same type can have different payoffs. We show analytically that for weak selection, this scenario is identical to the usual approach in which an individual interacts with the whole population. For strong selection, however, differences arise that are reflected in the fixation probabilities and lead to deviating evolutionary dynamics.     

Stochastic evolutionary dynamics of bimatrix games

Evolutionary game dynamics of two-player asymmetric games in finite populations is studied. We consider two roles in the game, roles α and β. α-players and β-players interact and gain payoffs. The game is described by a pair of matrices, which is called bimatrix. One's payoff in the game is interpreted as its fecundity, thus strategies are subject to natural selection. In addition, strategies can randomly mutate to others. We formulate a stochastic evolutionary game dynamics of bimatrix games as a frequency-dependent Moran process with mutation. We analytically derive the stationary distribution of strategies under weak selection. Our result provides a criterion for equilibrium selection in general bimatrix games.     

Mechanistic constraints and the unlikely evolution of reciprocal cooperation

Social evolution theory faces a puzzle: a gap between theoretical and empirical results on reciprocity. On the one hand, models show that reciprocity should evolve easily in a wide range of circumstances. On the other hand, empirically, few clear instances of reciprocity (even in a broad sense) have been found in nonhuman animals. In this paper, I aim to suggest and evaluate a novel reason concurring to solve this puzzle. I propose that it is difficult for reciprocity to evolve because it raises an evolutionary problem of bootstrapping: it requires that two complementary functions: (i) the ability to cooperate and (ii) the ability to respond conditionally to the cooperation of others, arise together and reach a significant frequency, whereas neither of them can be favoured in the absence of the other. I develop analytical models and simulations showing that, for this reason, the evolutionary emergence of reciprocal cooperation is highly unlikely. I then discuss the consequences of this result for our understanding of cooperation.     

The evolution of phenotypic traits in a predator-prey system subject to Allee effect

This paper considers the evolution of phenotypic traits in a community comprising the populations of predators and prey subject to Allee effect. The evolutionary model is constructed from a deterministic approximation of the stochastic process of mutation and selection. Firstly, we investigate the ecological and evolutionary conditions that allow for continuously stable strategy and evolutionary branching. We find that the strong Allee effect of prey facilitates the formation of continuously stable strategy in the case that prey population undergoes evolutionary branching if the Allee effect of prey is not strong enough. Secondly, we show that evolutionary suicide is impossible for prey population when the intraspecific competition of prey is symmetric about the origin. However, evolutionary suicide can occur deterministically on prey population if prey individuals undergo strong asymmetric competition and are subject to Allee effect. Thirdly, we show that the evolutionary model with symmetric interactions admits a stable limit cycle if the Allee effect of prey is weak. Evolutionary cycle is a likely outcome of the process, which depends on the strength of Allee effect and the mutation rates of predators and prey.     

Game theory and human evolution: a critique of some recent interpretations of experimental games

Economists and psychologists have been testing Nash equilibrium predictions of game theory models of human behavior. In many instances, humans do not conform to the predictions. These results are of great interest to biologists because they also raise questions about well-known ESS models of cooperation. Cooperation in certain one-shot, anonymous interactions, and a willingness to punish others at a net cost to oneself are some of the most intriguing deviations from standard theory. One proposed explanation for these results that is receiving increasing attention invokes the cultural group selection of 'other regarding' social norms. We critically review this explanation. We conclude that experimental results reveal limits in two implicit models of cognitive structure commonly employed by economists and evolutionary biologists.