Robust and Diverse Multi-Agent Learning via Rational Policy Gradient

AI summary

Overview

• Research area: Multi-agent reinforcement learning (MARL), adversarial optimization, and game theory — specifically how to keep adversarial training useful in cooperative and general-sum games.
• Technical level: Advanced. The paper assumes familiarity with partially-observable stochastic games, best-response functions, policy gradient methods, and opponent shaping.
• Scope (one sentence): The paper identifies a failure mode called "self-sabotage" in adversarial multi-agent optimization, proposes a formalism (RPO) and a gradient-based algorithm (RPG) that prevent it by forcing agents to remain rational, and empirically validates the approach across matrix games, Overcooked, STORM, and Hanabi.

What This Paper Is About

Adversarial optimization — training one agent to minimize another's reward — works well in zero-sum games, where self-play naturally produces robust policies. In cooperative or general-sum games, applying it naively pushes agents to "self-sabotage": instead of exposing useful weaknesses, an adversary simply refuses to cooperate or actively harms the shared reward, which halts useful learning. The paper's goal is to make adversarial optimization work in these settings by constraining every agent's policy to be a best response to some possible partner policy, then showing this constraint can be enforced with gradients.

Key Contributions

1. A new formalism, Rationality-preserving Policy Optimization (RPO): An adversarial optimization problem whose constraint requires an agent's policy to be optimal with respect to at least one policy the other agents might play — formalized as π_i ∈ BR(π'_{-i}) for some co-policy π'_{-i}.
2. A gradient-based deep learning algorithm, Rational Policy Gradient (RPG): RPG introduces a "manipulator" for each "base agent." Base agents only maximize their own reward in a copy of the game (their manipulator environment), while manipulators use opponent shaping to guide base agents toward the adversarial objective. Manipulators are discarded after training.
3. Five novel adversarial optimization algorithms built from RPG: AP-RPG, AT-RPG, PAIRED-RPG, PAIRED-Attack-RPG, and AD-RPG — covering adversarial example finding, robustification, and diversity.
4. Empirical validation in matrix games, Overcooked layouts, a modified STORM, and a simplified 2-player Hanabi, showing that RPG avoids self-sabotage and outperforms existing baselines.

Main Findings

• RPG finds meaningfully diverse policies (Claim 1): In the original STORM environment, AD-RPG's manipulators learn around episode 10000 to lead each base agent to gather a different colored coin, causing cross-play reward to drop while self-play continues to rise. In cross-play, the agents first gather opposite-colored coins, then adapt and gather a matching coin for partial reward. CoMeDi, by contrast, drives cross-play reward straight to zero. In a modified STORM where either agent occupying the top-left square yields a reward of −0.1 per timestep, CoMeDi learns to intentionally sabotage, while AD-RPG avoids the sabotage state in both self-play and cross-play.
• Overcooked diversity results are partly an illusion of sabotage: In the cramped room layout, CoMeDi reaches a self-play reward of 220 and a cross-play reward of 2, while AD reaches 240 self-play and 1.25 cross-play; AD-RPG reaches 240 in both. Inspection shows CoMeDi and AD agents stand in front of the plate dispenser to block their partner — sabotage, not genuine diversity. The paper reads the high AD-RPG cross-play score as evidence that cramped room contains very little genuine diversity.
• AD-RPG policies are more robust to differing partners (Claim 2): Cross-play grids were built from five seeds each for AD, AD-RPG, SP with entropy coefficient 0.01, and SP with entropy coefficient 0.05, with AD and AD-RPG using a population size of two policies; each grid square is the average reward over 1000 rollouts. AD-RPG performs nearly perfectly with any partner, including in forced coordination, where even the two sets of SP seeds are completely incompatible. The grid is asymmetric because the column player in forced coordination must catch onions passed by the partner, and AD-RPG adapts to any passing strategy. In the more complex coordination ring and counter circuit layouts, SP achieves very low cross-play rewards even with high entropy, while AD-RPG maintains much higher cross-play rewards. In 3-color and 4-color Hanabi, SP with high entropy achieves significantly higher intra-population rewards than low entropy.
• RPG finds non-trivial adversarial examples (Claim 3): In "unobserved STORM," where agents cannot observe their partner's position, PAIRED-A-RPG and AP-RPG both find weaknesses in fixed victim policies without sabotaging. Victim training reward and performance against AP (which trivially achieves 0.0 because it self-sabotages by collecting no coins), PAIRED-A-RPG, and AP-RPG respectively: PAIRED 0.13 / 0.0 / 0.50 / 0.42; PAIRED-RPG 0.93 / 0.0 / 0.84 / 0.85; AT 0.0 / 0.0 / 0.0 / 0.0; AT-RPG 0.65 / 0.0 / 0.72 / 0.88; AD 0.00 / 0.0 / 0.00 / 0.00; AD-RPG 0.98 / 0.0 / 0.25 / 0.96; Self-play 0.98 / 0.0 / 0.16 / 0.96.
• RPG prevents self-sabotage across algorithms (Claim 4): The non-RPG variants (AT, PAIRED, AD) fail during training due to self-sabotage, while their RPG counterparts train successfully.
• RPG generalizes adversarial training: AT-RPG is identical to AT in zero-sum games, making it a strict generalization.
• Illustrative matrix game: In the paper's 3×3 cooperative game with victim actions A and B and adversary actions C, D, and E (payoffs: A-C = 1, A-D = 0, A-E = −1; B-C = 0, B-D = 1, B-E = −1), a naive adversary always plays E to minimize the victim's reward at −1. Under RPO, the adversary plays D. The only equilibrium is for the victim to play a uniform mixture over A and B, guaranteeing a minimum expected reward of 0.5 against any rational co-policy.
• Partner-play regularization: Because base agents train against manipulators but are evaluated against other base agents, the authors add low-weight (ε-scaled) rollouts of each base agent paired with the base agents it is evaluated against, to limit distribution shift and prevent manipulators from exploiting out-of-distribution base behavior.

Methodology in Plain English

The authors start from a simple diagnostic: if you tell an agent "minimize my teammate's reward" in a cooperative game, the cheapest solution is often to break the game — refuse to cooperate or actively harm the shared outcome. That solution is not rational in the underlying game, since it lowers the agent's own reward too. So instead of optimizing the adversarial objective directly, RPO adds a constraint: any policy the agent learns must be a best response to at least one partner policy the other agent could plausibly play.

Enforcing that constraint directly in a single objective is hard, so RPG splits each agent into two pieces. A base agent simply plays the game normally, maximizing its own reward against a partner called a manipulator. The manipulator never plays the real game — it only chooses the partner the base agent trains against, and its goal is the original adversarial objective (for example, making two base agents perform poorly together). To influence the base agent, the manipulator uses opponent shaping: it takes higher-order gradients through the base agent's own gradient update, effectively reasoning about how its behavior changes what the base agent learns. The manipulator's loss uses Loaded DiCE, which preserves the higher-order gradient dependencies that ordinary RL surrogate losses do not. After training, manipulators are thrown away and the base agents are the solution. This design means agents can still be pushed toward adversarial outcomes, but only through policies that would be sensible responses to some real partner — which is what prevents sabotage. The paper notes RPG is agnostic to the underlying RL algorithm used for the gradient computations.

Why This Matters

• Impact on research: The paper reframes a known failure of adversarial optimization in cooperative games as a constraint-satisfaction problem and gives a general recipe for fixing it. Because RPG extends AP, AT, PAIRED, and AD without changing their zero-sum behavior, it can be applied broadly rather than as a one-off patch — and the authors describe fully eliminating self-sabotage in adversarial diversity algorithms as an open problem the prior literature had not solved.
• Real-world applications (these are categories of problems the method targets, not deployments reported in the paper):
  • Training robots or software agents that must cooperate with unfamiliar, possibly non-expert human partners.
  • Multi-agent logistics and coordination systems where teammates may change or be replaced.
  • Safety and robustness testing of deployed cooperative AI policies, via rational adversarial examples rather than degenerate ones.
  • Generating diverse strategy populations for game AI, simulation, and curriculum learning.
• Industry relevance: Any system that pairs learned agents with other agents — autonomous fleets, collaborative robotics, game NPCs, or agentic software that hands tasks between components — needs policies that adapt to varied partners. A method that fails by "sabotaging the task" is not deployable; one that always produces rational, best-response-consistent policies is far more directly usable. The paper provides public code and a project page with demos for interacting with the trained agents.

Future Directions

• Scaling beyond two players and small populations: The exposition limits itself to two-player games, and the experiments use a population size of two for AD and AD-RPG. Whether the approach holds for many agents and larger populations is not established in the provided content.
• Removing remaining assumptions: The paper notes that policies lack history-dependence in Hanabi and speculates this contributed to the unusual entropy results. Extending RPG to history-dependent policies with external memory is mentioned as possible but is not empirically demonstrated in the provided content.
• Closing the adversarial gap on diverse victims: In unobserved STORM, PAIRED-A-RPG attacks only reached 0.25 against AD-RPG and 0.16 against self-play victims, versus 0.84 and 0.96 against PAIRED-RPG. Understanding why some robust victims remain hard to attack is left open.
• Understanding robustness in richer layouts: The finding that cramped room contains little genuine diversity suggests that the environments used to benchmark diverse multi-agent learning may be too easy to distinguish real diversity from sabotage. Which environment families actually admit meaningful diverse solutions is an open question.

Target Audience

This paper is most useful for multi-agent RL researchers and graduate students working on robustness, opponent modeling, adversarial training, or diversity in cooperative and general-sum games. It also suits practitioners who build multi-agent systems and need policies that cooperate with partners they were not trained against, and researchers interested in opponent shaping and higher-order gradient methods. Readers without a background in MARL and game theory will find the formalism in Sections 2–4 demanding, though the motivating matrix game and the Overcooked/STORM sabotage examples make the core idea accessible.

Authors’ abstract

Adversarial optimization algorithms that explicitly search for flaws in agents' policies have been successfully applied to finding robust and diverse policies in multi-agent settings. However, the success of adversarial optimization has been largely limited to zero-sum settings because its naive application in cooperative settings leads to a critical failure mode: agents are irrationally incentivized to self-sabotage, blocking the completion of tasks and halting further learning. To address this, we introduce Rationality-preserving Policy Optimization (RPO), a formalism for adversarial optimization that avoids self-sabotage by ensuring agents remain rational--that is, their policies are optimal with respect to some possible partner policy. To solve RPO, we develop Rational Policy Gradient (RPG), which trains agents to maximize their own reward in a modified version of the original game in which we use opponent shaping techniques to optimize the adversarial objective. RPG enables us to extend a variety of existing adversarial optimization algorithms that, no longer subject to the limitations of self-sabotage, can find adversarial examples, improve robustness and adaptability, and learn diverse policies. We empirically validate that our approach achieves strong performance in several popular cooperative and general-sum environments. Our project page can be found at https://rational-policy-gradient.github.io.