Reinforcement Learning Explains Conditional Cooperation and Its Moody Cousin

Direct reciprocity, or repeated interaction, is a main mechanism to sustain cooperation under social dilemmas involving two individuals. For larger groups and networks, which are probably more relevant to understanding and engineering our society, experiments employing repeated multiplayer social dilemma games have suggested that humans often show conditional cooperation behavior and its moody variant. Mechanisms underlying these behaviors largely remain unclear. Here we provide a proximate account for this behavior by showing that individuals adopting a type of reinforcement learning, called aspiration learning, phenomenologically behave as conditional cooperator. By definition, individuals are satisfied if and only if the obtained payoff is larger than a fixed aspiration level. They reinforce actions that have resulted in satisfactory outcomes and anti-reinforce those yielding unsatisfactory outcomes. The results obtained in the present study are general in that they explain extant experimental results obtained for both so-called moody and non-moody conditional cooperation, prisoner’s dilemma and public goods games, and well-mixed groups and networks. Different from the previous theory, individuals are assumed to have no access to information about what other individuals are doing such that they cannot explicitly use conditional cooperation rules. In this sense, myopic aspiration learning in which the unconditional propensity of cooperation is modulated in every discrete time step explains conditional behavior of humans. Aspiration learners showing (moody) conditional cooperation obeyed a noisy GRIM-like strategy. This is different from the Pavlov, a reinforcement learning strategy promoting mutual cooperation in two-player situations.     

Network fluctuations hinder cooperation in evolutionary games

In this paper we study the influence of random network fluctuations on the behavior of evolutionary games on Barabási-Albert networks. This network class has been shown to promote cooperation on social dilemmas such as the Prisoner's Dilemma and the Snowdrift games when the population network is fixed. Here we introduce exogenous random fluctuations of the network links through several noise models, and we investigate the evolutionary dynamics comparing them with the known static network case. The results we obtain show that even a moderate amount of random noise on the network links causes a significant loss of cooperation, to the point that cooperation vanishes altogether in the Prisoner's Dilemma when the noise rate is the same as the agents' strategy revision rate. The results appear to be robust since they are essentially the same whatever the type of the exogenous noise. Besides, it turns out that random network noise is more important than strategy noise in suppressing cooperation. Thus, even in the more favorable situation of accumulated payoff in which links have no cost, the mere presence of random external network fluctuations act as a powerful limitation to the attainment of high levels of cooperation.     

Nodes Having a Major Influence to Break Cooperation Define a Novel Centrality Measure: Game Centrality

Cooperation played a significant role in the self-organization and evolution of living organisms. Both network topology and the initial position of cooperators heavily affect the cooperation of social dilemma games. We developed a novel simulation program package, called ‘NetworGame’, which is able to simulate any type of social dilemma games on any model, or real world networks with any assignment of initial cooperation or defection strategies to network nodes. The ability of initially defecting single nodes to break overall cooperation was called as ‘game centrality’. The efficiency of this measure was verified on well-known social networks, and was extended to ‘protein games’, i.e. the simulation of cooperation between proteins, or their amino acids. Hubs and in particular, party hubs of yeast protein-protein interaction networks had a large influence to convert the cooperation of other nodes to defection. Simulations on methionyl-tRNA synthetase protein structure network indicated an increased influence of nodes belonging to intra-protein signaling pathways on breaking cooperation. The efficiency of single, initially defecting nodes to convert the cooperation of other nodes to defection in social dilemma games may be an important measure to predict the importance of nodes in the integration and regulation of complex systems. Game centrality may help to design more efficient interventions to cellular networks (in forms of drugs), to ecosystems and social networks.     

The Wisdom of Networks: A General Adaptation and Learning Mechanism of Complex Systems

I hypothesize that re‐occurring prior experience of complex systems mobilizes a fast response, whose attractor is encoded by their strongly connected network core. In contrast, responses to novel stimuli are often slow and require the weakly connected network periphery. Upon repeated stimulus, peripheral network nodes remodel the network core that encodes the attractor of the new response. This “core‐periphery learning” theory reviews and generalizes the heretofore fragmented knowledge on attractor formation by neural networks, periphery‐driven innovation, and a number of recent reports on the adaptation of protein, neuronal, and social networks. The core‐periphery learning theory may increase our understanding of signaling, memory formation, information encoding and decision‐making processes. Moreover, the power of network periphery‐related “wisdom of crowds” inventing creative, novel responses indicates that deliberative democracy is a slow yet efficient learning strategy developed as the success of a billion‐year evolution. Also see the video abstract here: https://youtu.be/IIjP7zWGjVE .     

Heterogeneous aspirations promote cooperation in the prisoner's dilemma game

To be the fittest is central to proliferation in evolutionary games. Individuals thus adopt the strategies of better performing players in the hope of successful reproduction. In structured populations the array of those that are eligible to act as strategy sources is bounded to the immediate neighbors of each individual. But which one of these strategy sources should potentially be copied? Previous research dealt with this question either by selecting the fittest or by selecting one player uniformly at random. Here we introduce a parameter that interpolates between these two extreme options. Setting equal to zero returns the random selection of the opponent, while positive favor the fitter players. In addition, we divide the population into two groups. Players from group select their opponents as dictated by the parameter , while players from group do so randomly irrespective of . We denote the fraction of players contained in groups and by and , respectively. The two parameters and allow us to analyze in detail how aspirations in the context of the prisoner''s dilemma game influence the evolution of cooperation. We find that for sufficiently positive values of there exist a robust intermediate for which cooperation thrives best. The robustness of this observation is tested against different levels of uncertainty in the strategy adoption process and for different interaction networks. We also provide complete phase diagrams depicting the dependence of the impact of and for different values of , and contrast the validity of our conclusions by means of an alternative model where individual aspiration levels are subject to evolution as well. Our study indicates that heterogeneity in aspirations may be key for the sustainability of cooperation in structured populations.     

An Extended N-Player Network Game and Simulation of Four Investment Strategies on a Complex Innovation Network

As computer science and complex network theory develop, non-cooperative games and their formation and application on complex networks have been important research topics. In the inter-firm innovation network, it is a typical game behavior for firms to invest in their alliance partners. Accounting for the possibility that firms can be resource constrained, this paper analyzes a coordination game using the Nash bargaining solution as allocation rules between firms in an inter-firm innovation network. We build an extended inter-firm n-player game based on nonidealized conditions, describe four investment strategies and simulate the strategies on an inter-firm innovation network in order to compare their performance. By analyzing the results of our experiments, we find that our proposed greedy strategy is the best-performing in most situations. We hope this study provides a theoretical insight into how firms make investment decisions.     

Win-stay, lose-shift strategies for repeated games-memory length, aspiration levels and noise.

Win-stay, lose-shift, the principle to retain a successful action is a simple and general learning rule that can be applied to all types of repeated decision problems. In this paper I consider win-stay, lose-shift strategies with diverse memory sizes and strategies that adapt their aspiration levels, i.e. the payoff level considered as "success". I study their evolution for the Prisoner's Dilemma, as well as in a rapidly changing environment, where a randomly selected game is assigned to the players. For win-stay, lose-shift strategies with memory one the average payoffs are computed and their evolutionary stability is discussed. Using computer simulations I show that the win-stay, lose-shift strategies with longer memory are very successful both for the Prisoner's Dilemma, where cooperation dominates even for high noise levels, and the randomly assigned games, where the players achieve nearly the expected Pareto optimal payoffs. I discuss the impact of noise and show that the memory length of the players increases with the noise level. These results indicate that the win-stay, lose-shift principle is a very successful strategy in repeated games with noise.     

How is the equilibrium of continuous strategy game different from that of discrete strategy game?

Cooperation in the prisoner's dilemma (PD) played on various networks has been explained by so-called network reciprocity. Most of the previous studies presumed that players can offer either cooperation (C) or defection (D). This discrete strategy seems unrealistic in the real world, since actual provisions might not be discrete, but rather continuous. This paper studies the differences between continuous and discrete strategies in two aspects under the condition that the payoff function of the former is a linear interpolation of the payoff matrix of the latter. The first part of this paper proves theoretically that for two-player games, continuous and discrete strategies have different equilibria and game dynamics in a well-mixed but finite population. The second part, conducting a series of numerical experiments, reveals that such differences become considerably large in the case of PD games on networks. Furthermore, it shows, using the Wilcoxon sign-rank test, that continuous and discrete strategy games are statistically significantly different in terms of equilibria. Intensive discussion by comparing these two kinds of games elucidates that describing a strategy as a real number blunts D strategy invasion to C clusters on a network in the early stage of evolution. Thus, network reciprocity is enhanced by the continuous strategy.     

Gender Differences in Cooperation: Experimental Evidence on High School Students

The emergence of cooperation among unrelated human subjects is a long-standing conundrum that has been amply studied both theoretically and experimentally. Within the question, a less explored issue relates to the gender dependence of cooperation, which can be traced back to Darwin, who stated that "women are less selfish but men are more competitive". Indeed, gender has been shown to be relevant in several game theoretical paradigms of social cooperativeness, including prisoner''s dilemma, snowdrift and ultimatum/dictator games, but there is no consensus as to which gender is more cooperative. We here contribute to this literature by analyzing the role of gender in a repeated Prisoners'' Dilemma played by Spanish high-school students in both a square lattice and a heterogeneous network. While the experiment was conducted to shed light on the influence of networks on the emergence of cooperation, we benefit from the availability of a large dataset of more 1200 participants. We applied different standard econometric techniques to this dataset, including Ordinary Least Squares and Linear Probability models including random effects. All our analyses indicate that being male is negatively associated with the level of cooperation, this association being statistically significant at standard levels. We also obtain a gender difference in the level of cooperation when we control for the unobserved heterogeneity of individuals, which indicates that the gender gap in cooperation favoring female students is present after netting out this effect from other socio-demographics factors not controlled for in the experiment, and from gender differences in risk, social and competitive preferences.     

Network Physiology: How Organ Systems Dynamically Interact

We systematically study how diverse physiologic systems in the human organism dynamically interact and collectively behave to produce distinct physiologic states and functions. This is a fundamental question in the new interdisciplinary field of Network Physiology, and has not been previously explored. Introducing the novel concept of Time Delay Stability (TDS), we develop a computational approach to identify and quantify networks of physiologic interactions from long-term continuous, multi-channel physiological recordings. We also develop a physiologically-motivated visualization framework to map networks of dynamical organ interactions to graphical objects encoded with information about the coupling strength of network links quantified using the TDS measure. Applying a system-wide integrative approach, we identify distinct patterns in the network structure of organ interactions, as well as the frequency bands through which these interactions are mediated. We establish first maps representing physiologic organ network interactions and discover basic rules underlying the complex hierarchical reorganization in physiologic networks with transitions across physiologic states. Our findings demonstrate a direct association between network topology and physiologic function, and provide new insights into understanding how health and distinct physiologic states emerge from networked interactions among nonlinear multi-component complex systems. The presented here investigations are initial steps in building a first atlas of dynamic interactions among organ systems.     

Exploring the Morphospace of Communication Efficiency in Complex Networks

Graph theoretical analysis has played a key role in characterizing global features of the topology of complex networks, describing diverse systems such as protein interactions, food webs, social relations and brain connectivity. How system elements communicate with each other depends not only on the structure of the network, but also on the nature of the system''s dynamics which are constrained by the amount of knowledge and resources available for communication processes. Complementing widely used measures that capture efficiency under the assumption that communication preferentially follows shortest paths across the network (“routing”), we define analytic measures directed at characterizing network communication when signals flow in a random walk process (“diffusion”). The two dimensions of routing and diffusion efficiency define a morphospace for complex networks, with different network topologies characterized by different combinations of efficiency measures and thus occupying different regions of this space. We explore the relation of network topologies and efficiency measures by examining canonical network models, by evolving networks using a multi-objective optimization strategy, and by investigating real-world network data sets. Within the efficiency morphospace, specific aspects of network topology that differentially favor efficient communication for routing and diffusion processes are identified. Charting regions of the morphospace that are occupied by canonical, evolved or real networks allows inferences about the limits of communication efficiency imposed by connectivity and dynamics, as well as the underlying selection pressures that have shaped network topology.     

DILEMMAS OF PARTIAL COOPERATION

Related to the often applied cooperation models of social dilemmas, we deal with scenarios in which defection dominates cooperation, but an intermediate fraction of cooperators, that is, “partial cooperation,” would maximize the overall performance of a group of individuals. Of course, such a solution comes at the expense of cooperators that do not profit from the overall maximum. However, because there are mechanisms accounting for mutual benefits after repeated interactions or through evolutionary mechanisms, such situations can constitute “dilemmas” of partial cooperation. Among the 12 ordinally distinct, symmetrical 2 × 2 games, three (barely considered) variants are correspondents of such dilemmas. Whereas some previous studies investigated particular instances of such games, we here provide the unifying framework and concisely relate it to the broad literature on cooperation in social dilemmas. Complementing our argumentation, we study the evolution of partial cooperation by deriving the respective conditions under which coexistence of cooperators and defectors, that is, partial cooperation, can be a stable outcome of evolutionary dynamics in these scenarios. Finally, we discuss the relevance of such models for research on the large biodiversity and variation in cooperative efforts both in biological and social systems.     

Cooperation in two-person evolutionary games with complex personality profiles

We propose a theory of evolution of social systems which generalizes the standard proportional fitness rule of the evolutionary game theory. The formalism is applied to describe the dynamics of two-person one-shot population games. In particular it predicts the non-zero level of cooperation in the long run for the Prisoner's Dilemma games, the increase of the fraction of cooperators for general classes of the Snow-Drift game, and stable nonzero cooperation level for coordination games.     

Age-Dependent Evolution of the Yeast Protein Interaction Network Suggests a Limited Role of Gene Duplication and Divergence

Proteins interact in complex protein–protein interaction (PPI) networks whose topological properties—such as scale-free topology, hierarchical modularity, and dissortativity—have suggested models of network evolution. Currently preferred models invoke preferential attachment or gene duplication and divergence to produce networks whose topology matches that observed for real PPIs, thus supporting these as likely models for network evolution. Here, we show that the interaction density and homodimeric frequency are highly protein age–dependent in real PPI networks in a manner which does not agree with these canonical models. In light of these results, we propose an alternative stochastic model, which adds each protein sequentially to a growing network in a manner analogous to protein crystal growth (CG) in solution. The key ideas are (1) interaction probability increases with availability of unoccupied interaction surface, thus following an anti-preferential attachment rule, (2) as a network grows, highly connected sub-networks emerge into protein modules or complexes, and (3) once a new protein is committed to a module, further connections tend to be localized within that module. The CG model produces PPI networks consistent in both topology and age distributions with real PPI networks and is well supported by the spatial arrangement of protein complexes of known 3-D structure, suggesting a plausible physical mechanism for network evolution.     

Learning rules for social foragers: implications for the producer-scrounger game and ideal free distribution theory

In population games, the optimal behaviour of a forager depends partly on courses of action selected by other individuals in the population. How individuals learn to allocate effort in foraging games involving frequency-dependent payoffs has been little examined. The performance of three different learning rules was investigated in several types of habitats in each of two population games. Learning rules allow individuals to weigh information about the past and the present and to choose among alternative patterns of behaviour. In the producer-scrounger game, foragers use producer to locate food patches and scrounger to exploit the food discoveries of others. In the ideal free distribution game, foragers that experience feeding interference from companions distribute themselves among heterogeneous food patches. In simulations of each population game, the use of different learning rules induced large variation in foraging behaviour, thus providing a tool to assess the relevance of each learning rule in experimental systems. Rare mutants using alternative learning rules often successfully invaded populations of foragers using other rules indicating that some learning rules are not stable when pitted against each other. Learning rules often closely approximated optimal behaviour in each population game suggesting that stimulus-response learning of contingencies created by foraging companions could be sufficient to perform at near-optimal level in two population games.     

Criticality Is an Emergent Property of Genetic Networks that Exhibit Evolvability

Accumulating experimental evidence suggests that the gene regulatory networks of living organisms operate in the critical phase, namely, at the transition between ordered and chaotic dynamics. Such critical dynamics of the network permits the coexistence of robustness and flexibility which are necessary to ensure homeostatic stability (of a given phenotype) while allowing for switching between multiple phenotypes (network states) as occurs in development and in response to environmental change. However, the mechanisms through which genetic networks evolve such critical behavior have remained elusive. Here we present an evolutionary model in which criticality naturally emerges from the need to balance between the two essential components of evolvability: phenotype conservation and phenotype innovation under mutations. We simulated the Darwinian evolution of random Boolean networks that mutate gene regulatory interactions and grow by gene duplication. The mutating networks were subjected to selection for networks that both (i) preserve all the already acquired phenotypes (dynamical attractor states) and (ii) generate new ones. Our results show that this interplay between extending the phenotypic landscape (innovation) while conserving the existing phenotypes (conservation) suffices to cause the evolution of all the networks in a population towards criticality. Furthermore, the networks produced by this evolutionary process exhibit structures with hubs (global regulators) similar to the observed topology of real gene regulatory networks. Thus, dynamical criticality and certain elementary topological properties of gene regulatory networks can emerge as a byproduct of the evolvability of the phenotypic landscape.     

Population networks: a large-scale framework for modelling cortical neural networks

Artificial neural networks are usually built on rather few elements such as activation functions, learning rules, and the network topology. When modelling the more complex properties of realistic networks, however, a number of higher-level structural principles become important. In this paper we present a theoretical framework for modelling cortical networks at a high level of abstraction. Based on the notion of a population of neurons, this framework can accommodate the common features of cortical architecture, such as lamination, multiple areas and topographic maps, input segregation, and local variations of the frequency of different cell types (e.g., cytochrome oxidase blobs). The framework is meant primarily for the simulation of activation dynamics; it can also be used to model the neural environment of single cells in a multiscale approach. Received: 9 January 1996 / Accepted in revised form: 24 July 1996     

How structure determines correlations in neuronal networks

Networks are becoming a ubiquitous metaphor for the understanding of complex biological systems, spanning the range between molecular signalling pathways, neural networks in the brain, and interacting species in a food web. In many models, we face an intricate interplay between the topology of the network and the dynamics of the system, which is generally very hard to disentangle. A dynamical feature that has been subject of intense research in various fields are correlations between the noisy activity of nodes in a network. We consider a class of systems, where discrete signals are sent along the links of the network. Such systems are of particular relevance in neuroscience, because they provide models for networks of neurons that use action potentials for communication. We study correlations in dynamic networks with arbitrary topology, assuming linear pulse coupling. With our novel approach, we are able to understand in detail how specific structural motifs affect pairwise correlations. Based on a power series decomposition of the covariance matrix, we describe the conditions under which very indirect interactions will have a pronounced effect on correlations and population dynamics. In random networks, we find that indirect interactions may lead to a broad distribution of activation levels with low average but highly variable correlations. This phenomenon is even more pronounced in networks with distance dependent connectivity. In contrast, networks with highly connected hubs or patchy connections often exhibit strong average correlations. Our results are particularly relevant in view of new experimental techniques that enable the parallel recording of spiking activity from a large number of neurons, an appropriate interpretation of which is hampered by the currently limited understanding of structure-dynamics relations in complex networks.     

Cooperation enhanced by the ‘survival of the fittest’ rule in prisoner's dilemma games on complex networks

Prevalence of cooperation within groups of selfish individuals is puzzling in that it contradicts with the basic premise of natural selection, whereby we introduce a model of strategy evolution taking place on evolving networks based on Darwinian ‘survival of the fittest’ rule. In the present work, players whose payoffs are below a certain threshold will be deleted and the same number of new nodes will be added to the network to maintain the constant system size. Furthermore, the networking effect is also studied via implementing simulations on four typical network structures. Numerical results show that cooperators can obtain the biggest boost if the elimination threshold is fine-tuned. Notably, this coevolutionary rule drives the initial networks to evolve into statistically stationary states with a broad-scale degree distribution. Our results may provide many more insights for understanding the coevolution of strategy and network topology under the mechanism of nature selection whereby superior individuals will prosper and inferior ones be eliminated.