The Stackelberg Speaker: Optimizing Persuasive Communication in Social Deduction Games

AI summary

Overview

Research area: Multi-agent reinforcement learning and LLM-based social reasoning, specifically persuasive communication in social deduction games (SDGs).

Technical level: Advanced. The paper combines game theory (Stackelberg competition), reinforcement learning (GRPO), and LLM fine-tuning, and assumes familiarity with policy optimization and language model internals.

Scope: The paper proposes and evaluates a training framework that fine-tunes an LLM to rewrite another LLM's utterances so that the next player in a social deduction game responds in ways favorable to the speaker.

What This Paper Is About

Existing LLM agents in social deduction games such as Werewolf, Avalon, and One Night Ultimate Werewolf focus on deducing other players' hidden roles and picking strategies, but they largely ignore the ability to actually persuade other players. The authors argue that winning depends not only on making correct deductions but on convincing others to respond in line with your intent. Their goal is to train an agent that deliberately optimizes its utterances to shift the next player's likely responses toward outcomes that benefit the speaker.

Key Contributions

1. A game-theoretic formulation. The authors model each turn of dialogue in an SDG as a two-player Stackelberg competition, where the current speaker is the leader and the next player is the follower. This converts persuasion into a local, tractable optimization problem rather than a global equilibrium computation over free-form language.

2. A training framework that optimizes utterances for persuasive impact. The framework has three stages: intent identification (the leader names desired and undesired follower responses), impact measurement (scoring candidate utterances by how much they shift the follower's response probabilities), and strategy optimization.

3. A GRPO-based refinement method. An open-source LLM (a Refiner) is trained with Group Relative Policy Optimization to rewrite base utterances produced by an API-based backend LLM, using rewards computed by a frozen Measurer model rather than a separate critic or human preference data.

4. Empirical validation across four environments. The authors report experiments on three SDGs (Werewolf, Avalon, ONUW) plus the open-ended social simulation environment Sotopia, including a human-subject study.

Main Findings

• Consistent gains in Werewolf, Avalon, and ONUW. Across 500 matches per game, integrating the trained Refiner with existing baselines improved overall win rates. For example, "Ours + LSPO" reached a 44.7% overall Werewolf win rate versus 38.6% for LSPO alone, and "Ours + Strategist" reached 61.3% overall in Avalon versus 57.4% for Strategist alone.

• Gains on both cooperative and deceptive roles. In Werewolf, "Ours + LSPO" achieved a 29.1% Village win rate and an 84.2% Werewolf win rate, compared with 24.9% and 72.8% for LSPO. In Avalon, "Ours + Strategist" reached 78.5% for the Good side and 35.2% for the Evil side, versus 77.6% and 27.0% for Strategist.

• The method complements rather than replaces baselines. The paper emphasizes that larger improvements appear when the Refiner is combined with stronger baselines, indicating it adds a persuasive dimension on top of existing strategic behavior.

• Transfer to Sotopia. With MetaMind as the base agent, the Refiner raised the Goal score from 8.70 to 8.92 and Overall from 4.03 to 4.20 on the 450 standard testing tasks, and on the 90 hard tasks it raised Goal from 7.16 to 7.59 and Overall from 3.60 to 3.85. Scores are reported on a 0 to 10 Goal scale.

• Both reward terms matter, but asymmetrically. An ablation with 50 matches per setting showed the full reward outperformed single-objective variants. In Werewolf, the full method reached 52/88/70 (Village/Werewolf/Average), while Positive-Only reached 46/82/64 and Negative-Only reached 18/80/49, essentially matching the ReAct baseline of 18/80/49.

• K = 3 intent targets is a stable choice. Varying the number of desired and undesired responses K showed performance peaking and stabilizing around K = 3; larger values did not help, which the authors attribute to low-quality samples adding noise.

• Generalization to unseen backend LLMs. Testing on GPT-5 and Qwen3-14B without additional fine-tuning, the approach consistently improved performance across both, which the authors interpret as evidence the Refiner learns model-agnostic persuasive principles rather than mimicking training-model style.

• Strong human-play results. In a study with 16 volunteers (each completing 5 games with 1 human and 6 AI agents), "Ours + LSPO" achieved the highest win rate at 44.1% and received the fewest votes from human players at 0.39, compared with 0.54 for human players, 0.70 for LSA, and 1.17 for ReCon.

• A stated limitation. Training tends to increase utterance length, which the authors attribute to the model adding rhetorical detail to maximize reward.

Methodology in Plain English

The authors treat each speaking turn as a leader-follower encounter. The current speaker is the "leader"; the next speaker is the "follower." The leader's job is to choose words that make the follower more likely to say something helpful to the leader and less likely to say something harmful.

The pipeline works in four steps. First, an API-based backend LLM (for example GPT-5 or Gemini-2.5) reads the game rules, the visible game state, the dialogue history, and the speaker's hidden role, then identifies three desirable responses the next player could give and three undesirable ones. Second, the backend produces a normal "base utterance." Third, a smaller open-source LLM called the Refiner rewrites that base utterance. Fourth, a frozen model called the Measurer (Qwen2.5-72B-Instruct) estimates the probability of each candidate follower response, and the reward is the summed log-probability of the desired responses minus the summed log-probability of the undesired responses.

Because the follower's real backend is a closed API, the Measurer stands in as a proxy for the follower's response distribution; it does not actually play the game. The Refiner is trained with GRPO, which compares a group of sampled candidate utterances against each other, computes relative advantages from their rewards, and updates the policy without needing a separate value network.

Implementation specifics: the Refiner is a LoRA adapter of rank 16 on Qwen2.5-7B-Instruct, trained with n = 8 sampled candidates, clipping epsilon = 0.2, KL weight beta = 0.04, a learning rate of 1 × 10⁻⁶, 4 A800 GPUs, 3 epochs, and roughly 50 hours, producing one checkpoint per game. Training data came from self-play with three backend LLMs (GPT-4o, Gemini-2.5-Flash, Claude-3.5-Haiku): 500 game logs per SDG with 4,000 randomly selected training instances per game, and for Sotopia, 800 multi-turn dialogues across 410 training scenarios with 3,000 selected instances.

Why This Matters

Impact on research. The paper reframes persuasion in multi-agent language games as a local leader-follower optimization, avoiding the intractability of computing global equilibria in free-form language. It also shows that RL can operate directly on the natural-language utterance space rather than reducing communication to choosing among a small set of candidate actions, which is how several prior SDG agents are described.

Real-world applications:

• Negotiation assistants that craft proposals to steer a counterpart toward mutually acceptable terms.
• Customer-facing dialogue systems where the goal is to shift a user's stated intent toward a beneficial outcome.
• Collaborative planning tools that need to convince a human partner to adopt a particular plan or division of labor.
• Training or simulation environments for human communication skills, including conflict de-escalation and trust-building scenarios.

Industry relevance. The approach is modular: the Refiner is a lightweight adapter that can be attached on top of any existing agent or API-based model, and the authors demonstrate it works on unseen backends without retraining. That makes it deployable as an augmentation layer rather than a full replacement, which is attractive for organizations already using commercial LLM APIs.

The authors also flag ethical risk explicitly: enhanced persuasion could be repurposed for manipulation or social engineering, and they argue for strict safety guardrails.

Future Directions

• Controlling verbosity. The authors state that training lengthens utterances and propose adding length penalties or conciseness rewards to the optimization objective.

• Better follower models. The Measurer is a proxy for a closed-source follower. Replacing or calibrating it against real human response distributions is an open problem the setup invites.

• Beyond local turn-level optimization. The Stackelberg formulation optimizes one turn at a time; whether persuasive strategies should be planned across multiple turns or entire games remains open.

• Extending to more open-ended social settings. The paper demonstrates transfer to Sotopia, which raises the question of how far the framework generalizes to domains with weaker win/loss structure, such as the negotiation, cooperation, and competition scenarios it lists.

Target Audience

Researchers and practitioners working on LLM agents, multi-agent reinforcement learning, and strategic communication. It is most useful to readers already comfortable with policy-gradient methods and language model fine-tuning who are interested in social reasoning benchmarks. Those focused on game theory, human-AI interaction, or the safety of persuasive AI systems will also find the framing and the human-subject results relevant, though the paper is not written as an introductory text.

Authors’ abstract

Large language model (LLM) agents have shown remarkable progress in social deduction games (SDGs). However, existing approaches primarily focus on information processing and strategy selection, overlooking the significance of persuasive communication in influencing other players' beliefs and responses. In SDGs, success depends not only on making correct deductions but on convincing others to response in alignment with one's intent. To address this limitation, we formalize turn-based dialogue in SDGs as a Stackelberg competition, where the current player acts as the leader who strategically influences the follower's response. Building on this theoretical foundation, we propose a reinforcement learning framework that trains agents to optimize utterances for persuasive impact. Through comprehensive experiments across three diverse SDGs, we demonstrate that our agents significantly outperform baselines. This work represents a significant step toward developing AI agents capable of strategic social influence, with implications extending to scenarios requiring persuasive communication. Our code and data are available at https://3dagentworld.github.io/leader_follower.