DiffFP: Learning Behaviors from Scratch via Diffusion-based Fictitious Play

AI summary

Overview

Research area: Multi-agent reinforcement learning and game theory — specifically extending Fictitious Play (FP) to continuous state and action spaces using diffusion-based policy representations.

Technical level: Advanced. The paper assumes working familiarity with Markov games, Nash equilibria, exploitability, self-play, DDPM-style diffusion models, and off-policy actor-critic methods (SAC, TD3, Q-learning). The prose is readable, but the method section is dense with equations.

Scope (1 sentence): DiffFP is a self-play framework that trains diffusion policies from scratch to approximate best responses inside a (generalized weakened) Fictitious Play loop, and is evaluated on a multi-agent racing task and two Multi-Particle Environment (MPE) zero-sum tasks against SAC, TD3, and QSM baselines.

What This Paper Is About

Self-play reinforcement learning works well in discrete games like Go, Chess, and StarCraft II, but in continuous action spaces it is hard to compute an exact best response, and standard unimodal RL policies (e.g., Gaussian actors) tend to overfit to recent opponents, forget earlier strategies, and cycle. The goal of this paper is to build a Fictitious Play loop where the best response at each iteration is represented by a diffusion policy, so that agents learn diverse, multimodal, and less exploitable strategies from scratch in continuous, partially observable zero-sum games.

Key Contributions

1. DiffFP learns from scratch via iterative self-play, training a diffusion policy as the best response to an evolving average opponent strategy — no demonstrations or offline data needed.
2. Explicit online modeling of the average strategy. Rather than approximating the opponent's average policy with supervised learning, DiffFP keeps the average and updates it online, following the exact averaging rule of Fictitious Play and sampling from the pool of historical best responses weighted by their update coefficients. This is claimed to stabilize learning dynamics and reduce non-stationarity.
3. Sample-efficient multimodal action modeling. Diffusion policies capture multimodal action distributions in continuous action spaces, enabling diverse behaviors rather than the single dominant mode typical of unimodal policies.
4. Empirical validation of convergence to ε-Nash equilibria in continuous-space zero-sum games, with reported gains of up to 3× faster convergence and 30× higher success rates on average versus RL-based baselines.

Main Findings

• Faster, more stable exploitability convergence. In the racing task, DiffFP "consistently outperforms all baselines, exhibiting both faster and more stable convergence toward low exploitability." Exploitability is reported in Figure 3 as mean and standard deviation over 10 episodes per FP iteration; Figure 2 reports exploitability over 100 evaluation runs and shows DiffFP with the lowest exploitability and minimal variance.

• Baselines converge on metrics but fail behaviorally. TD3 agents fall into loop-like trajectories, circling without track progress, and SAC agents frequently violate track boundaries. The paper attributes this to Q-value mode-seeking behavior and limited expressiveness.

• QSM is unstable. Although diffusion-based and expressive, QSM suffers high instability from its reliance on accurate value estimates, which are noisy in non-stationary multi-agent settings, leading to noisy and non-convergent exploitability plots.

• Training progression shows emergent strategy. Trajectory samples across FP iterations show the blue agent initially trailing, then overtaking on the inside and forcing the red agent to brake, and near convergence both agents attacking, defending, and driving each other toward track boundaries. Time-to-goal improved from 80 to 45 steps under identical initial conditions.

• Head-to-head racing results (Table I): DiffFP recorded a mean attacking gap of 17.70 m, 12/20 successes, 1/20 crashes, and mean reward 0.92; QSMFP recorded 8.26 m, 3/20 successes, 4/20 crashes, and mean reward 0.80. The attacking agent is initialized behind the race leader.

• Robustness to unseen opponents in racing (Table II): against up to five previously unseen adversaries, DiffFP had 4/20 crashes and mean reward 0.756, versus QSMFP with 11/20 crashes and 0.556.

• MPE-Adversary cross-play (Table III): when DiffFP played ego, it beat SACFP 79–10 (11 draws), TD3FP 75–11 (14 draws), and QSMFP 75–12 (13 draws). When DiffFP played adversary: SACFP vs DiffFP was 62 ego wins (17↓), 16 adversary wins (6↑), 22 draws (11↑); TD3FP vs DiffFP was 50 (25↓), 18 (7↑), 32 (18↑); QSMFP vs DiffFP was 63 (12↓), 27 (15↑), 10 (3↓).

• MPE-Tag cross-play (Table IV): DiffFP as ego beat SACFP 81–19, TD3FP 89–11, and QSMFP 64–36. As adversary: SACFP vs DiffFP 75–25 (6↑), TD3FP vs DiffFP 66–34 (23↑), QSMFP vs DiffFP 68–32 (4↓). Also reported: QSMFP vs SACFP 66–34, and SACFP vs QSMFP 42–58 (24↑).

• Population-based hardening. Against SACPBT (SAC with Population-Based Training), which adds two more adversaries, the adversary role achieved a 90% improvement over SACFP against DiffFP. DiffFP without PBT performed comparably against SACPBT, avoiding the added computational cost.

• Where Q-value methods keep up. In MPE-Tag, baseline performance was comparable to DiffFP, particularly for adversaries. The paper explains this by the team-centralized setup: with one ego and three adversaries, the setting resembles a population-based hardening variant of a 1v1 predator-prey game, and Q-mode-seeking models tend to do well as adversaries because more samples reduce Q-value estimation error (visible in QSMFP's performance).

• Qualitative multimodal behavior. Only DiffFP and SACFP converged to near-optimal performance in MPE, and the Q-value map (Figure 4C) shows one ego agent acting as a decoy to distract the adversary while the other reaches the goal.

Methodology in Plain English

The researchers wrap a diffusion policy inside a Fictitious Play loop.

1. Fictitious Play outer loop. Each agent repeatedly computes a best response to the empirical average of its opponent's past strategies, then updates its own strategy as a weighted average of old and new. The paper uses a generalized weakened version, which permits approximate best responses and small perturbations as long as the step sizes shrink appropriately.

2. Best response via RL, not exact optimization. Exact best responses are intractable in continuous spaces, so an RL agent approximates them. Critically, because the opponent's average is stochastic and not explicitly known to the ego agent, DiffFP does not approximate it with supervised learning; it samples from the pool of historical best-response policies, weighted by their update coefficients.

3. The policy is a diffusion model. A DDPM-style forward process gradually noises an action from the replay buffer into a Gaussian prior; the learned reverse process (conditioned on the agent's observation) denoises a Gaussian sample back into an action. Training combines a conditional denoising score-matching loss with double Q-learning to guide policy improvement via action gradients, plus reward-weighing to prioritize higher-return samples.

4. Architecture and setup. Actor and critic are multilayer perceptrons (MLPs). Experience is collected into a replay buffer; the Q-function is updated with double Q-learning, the diffusion policy is trained with the denoising loss, and actions undergo Q-guided refinement for a set number of action-ascent steps. Multi-agent teams use centralized training with decentralized execution (CTDE).

5. Evaluation. Exploitability (Definition 3) is the main metric — how much an agent's return could be improved by switching to a best response — with normalized episodic cumulative reward as the payoff measure. Environments are a multi-agent racing task (general-sum, dense rewards, symmetric agents) and MPE-Adversary and MPE-Tag (asymmetric roles, sparse rewards, partial observability). Experiments ran in Python 3.8 on a 12-core CPU with an RTX A6000 GPU.

Why This Matters

Impact on research. Many multi-agent RL methods that look good on reward or exploitability curves are actually brittle: the paper's qualitative evidence shows TD3 circling the track and SAC leaving track boundaries despite converging metrics. DiffFP argues that expressiveness of the policy class — not just the training algorithm — determines whether self-play produces genuinely strategic, non-exploitable behavior. It also connects the diffusion-policy literature (largely offline and demonstration-based) to online, from-scratch multi-agent training, and shows that modeling the opponent's average strategy directly can remove the need for a supervised approximation step.

Real-world applications (domains indicated in the paper):

• Autonomous racing and competitive vehicle control, where overtaking, defending, and collision avoidance must be learned in continuous action space.
• Robotics, cited as a domain with continuous action spaces, partial observability, and imperfect information.
• Autonomous driving, including the semi-competitive/adversarial settings referenced in the related work (e.g., ARC).
• Adversarial multi-agent pursuit-evasion and coverage tasks, as represented by the MPE-Tag and MPE-Adversary environments.

Industry relevance. The claim of up to 3× faster convergence and 30× higher success rates matters where simulation is the bottleneck: fewer training iterations mean lower compute cost. The claim that DiffFP performs comparably to a population-based training variant without PBT also matters, because population methods scale poorly — diminishing returns as population size grows, and relearning policies from scratch each iteration becomes intractable in larger action spaces.

Future Directions

1. Scaling to more agents and larger action spaces. The paper notes that relearning policies from scratch each iteration hinders scalability to larger action spaces and that population methods show diminishing returns; whether DiffFP's diffusion best responses scale further is untested.
2. Combining with population-based training more systematically. SACPBT achieved a 90% improvement in the adversary role against DiffFP in MPE-Adversary, and DiffFP matched it without PBT. Whether combining DiffFP with PBT yields further gains, or just added cost, is left open.
3. Extending low-exploitability guarantees to harder game classes. The related work explicitly calls extending Counterfactual Regret Minimization to continuous-action games with high-dimensional state and action spaces an open challenge; DiffFP addresses continuous spaces but the paper does not report formal exploitability bounds.
4. Real-time feasibility of diffusion sampling. Diffusion policies require iterative denoising at every decision step; the paper does not report inference latency or throughput, which would matter for deployment on physical systems.

Target Audience

Researchers and graduate students in multi-agent reinforcement learning and game theory who work on self-play, Nash equilibrium approximation, and opponent robustness; practitioners applying policy learning to continuous-control competitive settings such as autonomous racing, driving, and robotics; and anyone interested in the intersection of generative diffusion models with online RL. Readers should already be comfortable with MDPs/POMGs, best responses, exploitability, and diffusion model mechanics — the paper is not an introductory read.

Authors’ abstract

Self-play reinforcement learning has demonstrated significant success in learning complex strategic and interactive behaviors in competitive multi-agent games. However, achieving such behaviors in continuous decision spaces remains challenging. Ensuring adaptability and generalization in self-play settings is critical for achieving competitive performance in dynamic multi-agent environments. These challenges often cause methods to converge slowly or fail to converge at all to a Nash equilibrium, making agents vulnerable to strategic exploitation by unseen opponents. To address these challenges, we propose DiffFP, a fictitious play (FP) framework that estimates the best response to unseen opponents while learning a robust and multimodal behavioral policy. Specifically, we approximate the best response using a diffusion policy that leverages generative modeling to learn adaptive and diverse strategies. Through empirical evaluation, we demonstrate that the proposed FP framework converges towards $ε$-Nash equilibria in continuous- space zero-sum games. We validate our method on complex multi-agent environments, including racing and multi-particle zero-sum games. Simulation results show that the learned policies are robust against diverse opponents and outperform baseline reinforcement learning policies. Our approach achieves up to 3$\times$ faster convergence and 30$\times$ higher success rates on average against RL-based baselines, demonstrating its robustness to opponent strategies and stability across training iterations