PopGen: Generating Semantically Diverse Populations for Zero-Shot Coordination

AI summary

Overview

Research area: Multi-agent reinforcement learning (MARL), specifically zero-shot coordination (ZSC) and automated population generation for cooperative agents.

Technical level: Intermediate. The paper assumes familiarity with reinforcement learning basics (policies, rewards, PPO) and cooperative benchmarks, but its central ideas — measuring behavioral diversity through language and turning text into executable code — are explained in an accessible way.

Scope: The paper introduces PopGen, an LLM-driven framework that generates semantically diverse partner agents for training cooperators that generalize to unseen teammates, and evaluates it across four cooperative environments against four population-play baselines.

What This Paper Is About

In cooperative multi-agent tasks, the same goal can be achieved through many different conventions, so an agent trained with one partner may fail completely with another that follows a different but equally valid strategy. A common fix is population play: train the agent (the "cooperator") against many partners so that test-time partners look familiar. The catch is that existing methods judge diversity from low-level trajectory statistics, which can count two different paths to the same strategy as "diverse" while missing genuinely different high-level play styles. PopGen instead asks an LLM to describe behavior in language, where strategic differences are explicit, and then turns those descriptions into runnable agent code.

Key Contributions

1. PopGen, a two-stage population generator. Stage 1 produces a diverse set of natural-language behavior descriptions for an environment; Stage 2 converts each description into an executable Python policy through an iterative, feedback-driven refinement loop.
2. Semantic diversity as the training signal. Diversity is measured as the average pairwise distance between unit-normalized text embeddings of behavior descriptions, rather than trajectory-, action-, or latent-space statistics.
3. A demonstrated zero-shot coordination gain. A cooperator trained against a PopGen population ("PopPlay") attains the highest mean team reward in all four evaluated environments compared with Self-Play, FCP, MEP, and CoMeDi, and the highest worst-case reward against held-out partners.
4. Ablations on population size, quality–diversity trade-off, and agent implementation. The paper shows semantic diversity and worst-case robustness scale with population size in Overcooked, compares PopGen against DIR, Verbalized Sampling, and SemDiv on a quality/diversity trade-off, and shows that "Policy Code" partners follow their intended behaviors more faithfully than partners trained on LLM-generated reward functions.

Main Findings

• Better generalization to unseen partners: PopPlay reaches the highest mean return in every environment. It clearly outperforms baselines in Level-Based Foraging (LBF) and Predator-Prey (PP), and exceeds the strongest baseline (CoMeDi) in Point-Mass Rendezvous (PMR) and Overcooked with overlapping error bars (mean and standard deviation over five runs).
• Better worst-case robustness: Worst-case team reward against the least compatible held-out partner was 1.41 ± 0.89 (PMR), 0.76 ± 0.16 (LBF), 1.33 ± 0.13 (PP), and 31.2 ± 44.6 (Overcooked) for PopPlay. Self-Play and FCP fall to 0.00 ± 0.00 in PMR and Overcooked worst-case.
• Role adaptation in Overcooked: The same PopPlay cooperator serves soup when its partner fetches onions, and fetches onions and cooks when the partner takes the dish. Its subtask share averaged over four held-out partners was 26% onion-to-pot, 26% start pot, 27% scoop soup, and 21% serve soup, whereas Self-Play and FCP completed 0% onion-loading and concentrated on scooping (56% and 62%) and serving (44% and 38%). CoMeDi was biased toward pot loading (68%).
• Diversity and robustness scale with population size: In Overcooked, using n ∈ {1, 2, 3, 4, 6, 8} distinct agents at a fixed budget of 8 partners, semantic diversity rose from 0 at n = 1 to 0.51 at n = 4 and then saturated. Adding the 4th agent gave a +34% performance gain with held-out partners. For n ≤ 2 every cooperator failed completely with at least one held-out partner, while from n = 3 onward worst-case reward jumped, up to 39 on average.
• Quality is preserved while diversity is pushed: Over 30 generated behaviors per environment, PopGen reached diversity 0.728 with quality 12.74, versus Verbalized Sampling at 0.726 diversity and 12.03 quality (a 6% quality improvement at comparable diversity). Against SemDiv, PopGen gives up 6% of quality for a 30% gain in diversity. DIR had the highest quality (14.45) but the lowest diversity (0.446), making PopGen 63% more diverse.
• Policy code beats reward functions for instantiating behaviors: With an identical set of textual behaviors, code-based partners adhered to their target behaviors at 0.66 ± 0.07 versus 0.43 ± 0.07 for reward-function partners. The gap was largest in discrete environments: LBF 0.61 vs 0.13, Overcooked 0.63 vs 0.41. Cooperators trained with Policy Code partners achieved higher test-time reward in all four environments.
• Mode collapse in baselines: Self-Play and FCP populations predominantly focused on onion fetching, so their cooperators rarely needed to fetch onions themselves and lacked that behavior with held-out partners.

Methodology in Plain English

The pipeline has two stages, both driven by large language models.

Stage 1 — inventing distinct play styles. The environment description is given to an LLM, which is asked for many candidate behaviors. Because asking for variety directly tends to produce either repetitive or nonsense answers, the system generates far more candidates than it needs, then re-queries the model to keep the highest-quality ones. It also asks the model to state each candidate's probability alongside it (a trick borrowed from Verbalized Sampling). Quality is scored by a separate LLM judge on three axes — realism, relevance, and specificity — each rated 0 to 5 and summed to a total out of 15.

Stage 2 — turning descriptions into agents. Each behavior description is translated into a Python policy rather than a reward function or a live LLM controller, which keeps execution cheap and avoids training a separate RL agent per behavior. To check whether a policy really behaves as described, the system rolls it out with a fixed scripted partner across multiple initial states and asks a vision-language model to judge from screenshots and state sequences whether each trajectory matches the description. This yields an adherence score between 0 and 1. If adherence is below a threshold, a separate LLM advisor reads the judge's rationales and writes targeted feedback, and the generator tries again — up to a fixed number of iterations. If nothing reaches the threshold, the best-scoring candidate is kept.

Training the cooperator. The resulting partners are frozen and used as the population for PPO-based population play, with partners sampled using a rank-based prioritized scheme (from MEP) so the cooperator spends more time on partners it currently handles poorly.

Evaluation. Cooperators are tested against held-out partners that are fixed scripted agents with distinct behaviors, specified in advance without LLMs and never shown to the LLMs, so the generator cannot be biased toward the test set.

Why This Matters

The paper reframes diversity for multi-agent training as a semantic property that can be expressed and generated in language, rather than a geometric property of action or state trajectories. That shift matters because trajectory diversity can be increased without increasing the range of strategies an agent must learn to complement — the failure mode the authors demonstrate in the Self-Play and FCP baselines, which collapse onto onion fetching. It also shows a practical route to building training populations without hand-designing scripted bots or maintaining a live pool of RL agents, since code policies add negligible overhead to the training loop.

Real-world applications suggested by the setting:

• Human–robot collaboration in households or warehouses, where a robot must adapt to a person's preferred division of labor without prior coordination.
• Assistive and service robotics, where different users follow different but equally valid routines and the robot must infer and complement the human's convention.
• AI teammates in cooperative video games, matching the Overcooked-style evaluation used here.
• Multi-robot logistics and task allocation, where agents must agree on conventions such as meeting points or hand-off roles.
• Dialogue and negotiation agents that must work with users or systems holding unfamiliar conventions.

Industry relevance: the method requires only an environment description and API access to a capable LLM and VLM, so it is a relatively low-engineering path to producing diverse, interpretable training partners for cooperative systems. The finding that executable code outperforms LLM-generated reward functions for behavior fidelity is directly actionable for teams already using LLMs to author reward functions.

Future Directions

• Human studies. All evaluation partners are synthetic. The authors note that prior work reports transfer from human-like populations to real humans, but confirming that PopGen's semantic diversity improves human–AI coordination remains future work.
• Online adaptive behavior generation. Moving behavior generation inside the cooperator's training loop so PopGen can target the cooperator's current weaknesses, rather than generating a fixed population up front.
• Better context engineering. The refinement loop is bounded by the context given to the LLM; improved context-engineering techniques and stronger code LLMs and coding agents offer a direct route to better advisor feedback and better generated policies.
• Open question: run-to-run variability. Because both behavior and policy generation depend on LLMs, the resulting populations — and therefore cooperator performance — may vary across runs. The reported results average over five runs, but the sensitivity to model choice and prompting is not fully characterized.

Target Audience

Researchers and practitioners in multi-agent reinforcement learning, zero-shot coordination, and LLM-based agent generation, particularly those building training populations or synthetic teammates. It is also relevant to engineers applying LLMs to author agent behavior — for example through code generation versus reward-function generation — and to robotics teams that need cooperative policies robust to unfamiliar human or machine partners. Readers without RL background will follow the high-level argument but will need familiarity with population play and policy optimization to interpret the results tables.

Authors’ abstract

In multi-agent reinforcement learning, exposing a cooperator to partners with diverse strategies during training is key to better collaboration at test time. However, existing methods typically rely on trajectory-level proxies to measure diversity, which may label two distinct trajectories of the same strategy as diverse. Instead, our key insight is to leverage LLMs' ability to describe behaviorally distinct agents in environments, similar to how humans can recognize high-level strategies. To this end, we introduce PopGen, a framework that synthesizes semantically diverse agents from scratch using LLMs. PopGen first generates a diverse set of textual descriptions of agent behaviors, then translates each behavior into a code policy that can be executed within the environment. We show that cooperators trained against PopGen populations achieve robust zero-shot cooperation with unseen partners, outperforming population-play baselines across four collaborative environments. We further show that semantic diversity scales with PopGen population size, improving the cooperator's worst-case performance, and we identify design choices that preserve population quality while maximizing diversity. Our code is available at https://github.com/labicon/PopGen.