Inclusive Fitness as a Key Step Towards More Advanced Social Behaviors in Multi-Agent Reinforcement Learning Settings

AI summary

Overview

Research area: Multi-agent reinforcement learning (MARL), evolutionary game theory, and open-ended learning environments.

Technical level: Intermediate. The paper assumes some familiarity with reinforcement learning, game theory payoff matrices, and basic population-genetics concepts, but the core ideas are explained in accessible terms.

Scope: The paper proposes a multi-agent reinforcement learning framework in which each agent is assigned an abstract genotype and rewarded according to an inclusive-fitness-style "inclusive reward," then tests this reward in two network-based prisoner's dilemma experiments and outlines a planned extension to the Neural MMO environment.

What This Paper Is About

Most multi-agent reinforcement learning setups treat agents as either pure competitors or members of predefined teams, which produces only binary cooperation or competition. The authors argue that real organisms — including humans — sit on a continuous spectrum of cooperation that depends on genetic relatedness. They therefore build a reward function, inspired by inclusive fitness and Hamilton's rule from biology, that weights other agents' payoffs by how genetically similar they are to the learner, and they test whether this produces more varied and stable social behavior.

Key Contributions

1. An abstract genetics layer for MARL. Each agent is given a genotype: a sequence of n genes, where each gene locus k in [1, n] holds a variant, and different integer values represent different variants. Relatedness between two agents is quantified as a "Hamming similarity" between 0 and 1, derived from the normalized Hamming distance (with Damerau-Levenshtein distance suggested for genotypes of unequal length).

2. A formal inclusive reward function. The reward of agent i is defined as the sum over all agents j of their individual rewards weighted by the Hamming similarity between genotype i and genotype j. This lets helping genetic relatives be rewarded even though their payoffs are separate.

3. An analytical link to Hamilton's rule. For a general prisoner's dilemma with benefit b and cost c, the authors derive the condition c < hb, which they show is equivalent to Hamilton's rule, and they demonstrate that a prisoner's dilemma between two agents with closely matching genotypes effectively becomes a harmony game in which mutual cooperation is the only Nash equilibrium.

4. A roadmap for inclusive rewards in open-ended environments. The paper proposes three candidate reward functions — longevity, replication, and combined — for spatially and temporally extended Markov games such as Neural MMO, and reasons about the non-team-based social dynamics these could produce.

Main Findings

• Cooperation tracks genetic similarity under opponent discrimination. On a fully connected network where each agent can recognize which opponent it faces, cooperation appears at a Hamming similarity threshold that shifts with the cost-benefit ratio c/b, matching Hamilton's rule. Identical experiments run without inclusive rewards led to all defection.

• Inclusive rewards outperform individual rewards under limited dispersal. In random partition networks with community structure, the inclusive reward produced higher proportions of cooperation than individual rewards, even though some cooperation emerged without inclusiveness.

• Small dispersal and large benefit-to-cost ratios favor cooperation. Lower dispersal coefficients η (defined as p_out/p_in) and higher b/c ratios both led to higher levels of cooperation, consistent with limited dispersal theory.

• Mutation and selection create a moving target for learners. Because the population of genotypes changes over time, the level of cooperation between any two agents is determined by the size and content of the current genotype population, which the authors describe as adding a social dimension to the multi-agent autocurriculum.

• Non-team-based social structures become possible. Weighting cooperation by relatedness allows configurations a team-based scheme cannot express — for example, one agent cooperating with two others that are adversarial toward each other. The authors sketch a concrete scenario where type A shares h = 0.6 with types B and C, while B and C share only h = 0.2, which under a resource-scarcity threshold of h > 0.5 makes B and C adversaries while A remains cooperative with both.

• Cooperation is a product of genotype content, not a static label. The paper frames cooperation as continuous and time-varying rather than a fixed team assignment, and the authors hypothesize this drives an arms race of strategies in which each new strategy is a gradual improvement over other agents' earlier adaptations.

Methodology in Plain English

The authors treat genes as abstract bookkeeping: an agent's genotype is a list of numbers, and two agents are "related" to the extent that their lists match. Relatedness is measured with a similarity score between 0 and 1 built from the Hamming distance — essentially, counting how many positions differ and converting that into a match fraction.

They then change what agents are rewarded for. Instead of each agent maximizing only its own payoff, each agent maximizes its own payoff plus the payoffs of every other agent, each discounted by how genetically similar that other agent is. An agent always cares most about itself (similarity 1), and cares progressively less about distant relatives.

Two network experiments test this. In the first, 64 agents (one per unique genotype, since genotypes are length 6 with 2 variants per locus) sit on a fully connected network and can tell opponents apart, each maintaining a separate Q-table entry per opponent. Every time step, each agent plays a prisoner's dilemma against every opponent simultaneously, including itself. In the second, agents cannot tell opponents apart and instead sit on random partition networks with community structure, where agents inside a community connect with probability p_in and agents across communities with p_out; each community corresponds to one genotype. Average network degree is pinned at ⟨k⟩ = 9 so that changing dispersal does not simultaneously change connectivity. In both experiments, Q-learners optimize their myopic inclusive reward with a bandit-like discount factor of 0 and use ε-greedy exploration with exponential decay.

For the planned open-ended work, the authors intend to replace tabular Q-learning with PPO and LSTM layers, potentially using parameter sharing where one network is conditioned on a genotype identifier. In that design, the game starts with a single agent that can reproduce (passing 1/4 of its health and resources to offspring), and each gene can mutate to another variant with probability μ, creating new species with new policy identifiers.

Why This Matters

Impact on research. The paper offers an alternative to the binary team-versus-team structure that dominates multi-agent reinforcement learning research on autocurricula. By grounding the reward signal in a biologically motivated notion of genetic relatedness, it connects MARL directly to Hamilton's rule and limited dispersal theory, and it suggests that continuous, shifting cooperation levels — rather than fixed team membership — could let agents keep generating new strategic challenges for one another.

Real-world applications (as framed by the biologically inspired mechanism):

• Multi-agent training environments where fixed team labels are a poor fit, such as populations of agents that form and dissolve alliances over time.
• Resource-constrained simulation settings, where scarcity creates tension between helping close relatives and distant ones, a dynamic the authors explicitly model as a non-stationary spectrum of cooperation.
• Open-ended game AI research built on platforms such as Neural MMO, which the authors describe as open-source and customizable for agent configurations, reward functions, environment layout, and resources.
• Population and ecosystem modeling more broadly, since the reward framework is presented as applicable beyond Neural MMO to multi-agent reinforcement learning settings where non-stationary coalitions and expressive social dynamics matter.

Industry relevance. The methods discussed — tabular Q-learning for the network experiments and PPO with LSTM layers for the planned extension — are standard tooling, and the parameter-sharing scheme the authors propose is described as a common strategy in multi-agent RL, so the approach is designed to fit existing training pipelines. That said, the paper reports no industrial deployment, benchmarking against commercial systems, or computational cost figures.

Future Directions

• Move from matrix games to Markov games. The authors plan to test the framework in temporally and spatially extended environments, chiefly Neural MMO, where agents must learn sequences of low-level actions to realize high-level strategies and where credit assignment becomes difficult.

• Compare the three proposed reward functions. Longevity reward (Eq. 5) rewards an agent for every unique genotype alive, weighted by Hamming similarity; replication reward (Eq. 6) rewards births and penalizes deaths, again similarity-weighted; combined reward (Eq. 7) sums Hamming similarity over all living agents, capturing how many copies of each unique genotype exist. The authors note the replication reward equals the difference in the combined reward across two time steps, and state they intend to try all three and study their properties.

• Test the autocurriculum hypothesis. The central open question is whether inclusive rewards can sustain a multi-agent autocurriculum that produces increasingly complex and socially intelligent strategies, and whether the resulting spectrum of cooperation shifts as population size approaches the environment's carrying capacity.

• Explore genes that express agent properties. So far genes influence behavior only through the reward function, but the authors raise the possibility of genes that encode in-game statistics such as maximal health or combat strength.

• Separate the framework from earlier team-based work. The authors position their approach against Abrantes et al. (2020), whose reward function corresponds to their combined reward (Eq. 7), arguing that earlier results do not yet provide a clear existence proof for novel, non-team-based dynamics.

Target Audience

This paper is best suited to reinforcement learning researchers working on multi-agent systems, autocurricula, and open-ended learning; to researchers in evolutionary game theory and agent-based modeling who want to see biological principles such as Hamilton's rule and limited dispersal translated into a learnable reward signal; and to graduate students or advanced undergraduates who already understand basic RL and game-theoretic payoff matrices and want an example of how ideas from biology can be formalized into a machine learning objective.

Authors’ abstract

The competitive and cooperative forces of natural selection have driven the evolution of intelligence for millions of years, culminating in nature's vast biodiversity and the complexity of human minds. Inspired by this process, we propose a novel multi-agent reinforcement learning framework where each agent is assigned a genotype and where reward functions are modelled after the concept of inclusive fitness. An agent's genetic material may be shared with other agents, and our inclusive reward function naturally accounts for this. We study the resulting social dynamics in two types of network games with prisoner's dilemmas and find that our results align with well-established principles from biology, such as Hamilton's rule. Furthermore, we outline how this framework can extend to more open-ended environments with spatial and temporal structure, finite resources, and evolving populations. We hypothesize the emergence of an arms race of strategies, where each new strategy is a gradual improvement over earlier adaptations of other agents, effectively producing a multi-agent autocurriculum analogous to biological evolution. In contrast to the binary team-based structures prevalent in earlier research, our gene-based reward structure introduces a spectrum of cooperation ranging from full adversity to full cooperativeness based on genetic similarity, enabling unique non team-based social dynamics. For example, one agent having a mutual cooperative relationship with two other agents, while the two other agents behave adversarially towards each other. We argue that incorporating inclusive fitness in agents provides a foundation for the emergence of more strategically advanced and socially intelligent agents.