Stop Training Against Soft Users: Mentalized Adversarial Self-Play for Proactive Dialogue

AI summary

Overview

Research area: Natural Language Processing — specifically multi-turn proactive dialogue (persuasion, emotional support, empathy, negotiation) and the reinforcement-learning / self-play methods used to train agents for it.

Technical level: Intermediate. The paper assumes familiarity with LLM fine-tuning, reinforcement learning from grouped rollouts (GRPO), reference-policy KL regularization, and the standard practice of training dialogue agents against user simulators. The BDI (belief–desire–intention) mental-state framing is explained from first principles.

Scope: The paper introduces MASP, a three-phase training framework that casts proactive dialogue as a partially observable game between a system policy and a trainable, rationally resisting user policy over a shared latent BDI mental state, and evaluates it on four dialogue benchmarks against SFT and Planner-only baselines.

What This Paper Is About

Proactive dialogue agents are usually trained and tested against user simulators that are fixed, hand-written, or overly agreeable, so a high simulated success rate can reflect a "soft user bias" rather than genuine strategic ability. MASP's goal is to make robustness to a rationally resisting user part of the training objective instead of only an evaluation condition, by jointly training the system and the user as two adversarial agents that each see a different view of the user's hidden mental state. The paper also argues that conventional success-rate scoring hides this problem, and introduces a rationality-constrained success metric to expose it.

Key Contributions

1. A BDI-based latent mental state that evolves during training. To the authors' knowledge, MASP is the first multi-turn dialogue training framework to explicitly model belief, desire, and intention states and let those states change across turns, combining free-form BDI text with three scalar affective variables (receptivity, commitment/confidence, valence).

2. Mentalized adversarial self-play with a teacher/student split. The paper states it is the first to apply self-play to a multi-turn dialogue training framework. A frozen teacher mentalizer defines the latent state used for reward computation, while the system policy only observes a student estimate inferred from dialogue history, so the policy must learn to infer the user rather than exploit the reward-side state estimator.

3. Rationality-constrained users rather than unconditional adversaries. The user policy is rewarded for resisting unjustified state change and for behaving coherently (via a fixed rationality judge signal), which prevents collapse into either always-rejecting or always-agreeing simulators.

4. A new evaluation metric, RCSR (rationality-constrained success rate), plus hard-user evaluation. RCSR@T counts a dialogue as successful only if the task is completed and the average rationality of the successful prefix clears a threshold, filtering out successes caused by simulator sycophancy or unjustified concessions.

Main Findings

• P4G (persuasion), Qwen3-4B: MASP reaches SR 0.859, AT 4.08, RCSR 0.556 against the soft user and SR 0.788, AT 4.56, RCSR 0.566 against the hard user, versus Planner-only (0.677 / 4.88 / 0.434 soft; 0.616 / 5.55 / 0.384 hard) and SFT (0.616 / 6.08 / 0.384 soft; 0.576 / 6.23 / 0.455 hard).

• P4G, Qwen3-0.6B: MASP leads on all six reported cells — soft 0.707 / 5.14 / 0.465 and hard 0.626 / 5.76 / 0.434 — against SFT (0.414 / 6.99 / 0.253; 0.354 / 7.35 / 0.242) and Planner-only (0.424 / 6.32 / 0.232; 0.364 / 6.84 / 0.202).

• ESConv (emotional support): MASP reaches near-ceiling SR (0.990 / 1.96 / 0.949 soft and 1.000 / 1.88 / 0.964 hard on Qwen3-4B; 1.000 / 2.11 / 0.949 and 1.000 / 1.65 / 0.928 on Qwen3-0.6B), with the largest gap over SFT (0.467 / 7.17 / 0.410 soft). The authors explicitly caution that this saturation reflects the prior where somewhat_better counts as success, so they treat ESConv mainly as transfer and efficiency evidence.

• CraigslistBargain (negotiation): MASP leads on SR and AT for both backbones, including the buyer-side hard setting (0.621 / 5.61 / 0.259 on Qwen3-4B). One exception is Qwen3-0.6B soft RCSR, where SFT slightly exceeds MASP (0.057 vs 0.030), described as a small speed–rationality tradeoff. The very low Planner-only 0.6B soft SR (0.172) is explained as the fixed planner scaffold failing to close against a seller whose concessions do not match its hand-written strategy.

• EmpatheticDialogues (empathy): Absolute SR is low because success requires a strict resolved outcome, but MASP is still best in every cell (0.092 / 7.46 / 0.059 soft and 0.072 / 7.57 / 0.048 hard on Qwen3-4B; 0.063 / 7.64 / 0.044 and 0.036 / 7.78 / 0.024 on Qwen3-0.6B).

• Gains appear early, not as late recovery: On the hard-user P4G split with Qwen3-4B, MASP reaches RCSR@3 of 0.444 and RCSR@4 of 0.485, versus 0.212 / 0.273 for Planner-only and 0.071 / 0.141 for SFT, while retaining SR@8 of 0.788.

• Robustness across three user simulators: MASP's average SR is 0.795 and worst-case SR 0.737; average RCSR 0.562 and worst-case 0.556. Planner-only falls to 0.660 / 0.616 and 0.414 / 0.384; SFT to 0.579 / 0.545 and 0.407 / 0.384.

• Ablations show gains come from both training and inference scaffold: Removing the adaptive close injection drops SR by 15.2 points under the soft user and 8.1 points under the hard user. Removing the warmup policy produces the lowest AT (3.83 soft, 4.55 hard) yet collapses RCSR from 0.556 to 0.283 and from 0.566 to 0.283 — evidence of fast but poorly grounded "shortcut" successes.

• Human audits support the synthetic supervision: Across 160 dialogue prefixes, BDI text fields score 4.55–4.62 out of 5 with 91.9–94.4% at least 4; scalar fields show MAE 0.071–0.094 and Spearman 0.79–0.84. The rationality judge agrees with humans on 91.8% of 425 P4G turns (κ = 0.801), with 91.2% mean turn agreement over 100 dialogues and ρ = 0.81.

• Statistical reliability: With 10,000 paired bootstrap resamples on the 99-session hard-user P4G evaluation, MASP's improvements over Planner-only have 95% intervals excluding zero for SR, AT, and RCSR; against SFT the intervals exclude zero for SR and AT, but the RCSR interval [−0.0202, 0.2424] includes zero. Of 21 unsuccessful sessions, 20 contain at least three refusal labels.

• Comparison to prior P4G systems: MASP (Qwen3-4B) reports AT 4.08 and SR 0.859, versus LDPP (Qwen1.5-1.8B) at 5.57 / 0.795 and UDP (GPT-4o mini) at 7.71 / 0.598 — a 6.4-point SR gain and 1.49 fewer turns than LDPP, and 26.1 points / 3.63 turns versus UDP.

• Self-play learning dynamics are real, not scaffold artifacts: Over three P4G outer iterations, rollout SR rises 0.450 → 0.550 → 0.713 and reward 0.826 → 0.946 → 1.209, showing the system policy improving against a changing opponent.

• Hyperparameters and mechanics worth noting: Discount factor γ = 0.97 for all self-play runs; numerical stabilizer ε_prog = 10⁻⁶; evaluation turn budget T = 8; RCSR rationality threshold τ_q = 0.70; the BDI text encoder is a frozen Qwen3 backbone with mean-pooled, L2-normalized final hidden states.

Methodology in Plain English

The researchers treat a proactive conversation as a two-player game with hidden information.

1. Describing the user's mind. Instead of a static persona label, the user's state at each turn is written as a small structured record: three short natural-language sentences for what the user believes, wants, and intends to do, plus three numbers on fixed scales for how receptive, how committed/confident, and how positive or negative the user feels. The text pieces are turned into vectors by a frozen language-model encoder, and the three numbers are appended with weighting so the whole thing becomes a single point in a "mental state space."

2. Two mentalizers, deliberately unequal. A teacher mentalizer reads the dialogue history plus optional profile and goal to produce the "true" latent state; it is trained on machine-generated (silver) annotations and then frozen. A student mentalizer sees only the dialogue history. The system policy is conditioned on the student's estimate, while rewards are computed from the teacher's state. This asymmetry is deliberate: the policy cannot simply read off the reward model's internal state, so improving the dialogue requires actually learning to infer the person.

3. Measuring progress relative to where each user started. Each task has a target mental-state template written by hand (e.g., a receptive user who intends to donate, or reduced distress). Because users start in different places, the method does not measure raw distance to the target. It projects the current state onto the line running from that user's own starting state to the target, producing a normalized progress score that changes every turn — a dense signal even when task success is sparse.

4. Opposing rewards. The system is rewarded for moving the user along that progress direction minus a penalty for misreading the user's current state. The user is rewarded for the mirror-image term — blocking unjustified movement — plus a rationality term from a fixed judge that labels each user reply as rational (+1), irrational (−1), or unparseable/failed (0). A user that rejects everything scores badly; a user that caves for no reason also scores badly.

5. Training in three phases. Phase 0 trains the teacher and student mentalizers from silver annotations. Phase 1 uses behavior cloning to warm-start both the system and user policies and freezes copies as reference policies. Phase 2 alternates: freeze the user and reward-side modules, roll out dialogues, and update the system with GRPO-style grouped advantages and a token-level KL penalty to its reference; then freeze the system and update the user the same way against its own reference; then refresh the student mentalizer on self-play prefixes mixed with real dialogue prefixes. Initial user states are sampled from a stored pool of initial BDI states (a "BDI prior"), and continuous student estimates are rendered back into text by nearest-neighbor retrieval.

6. Evaluation. Each episode starts from a held-out context and runs system utterance → user reply → append to history, until success or eight exchanges. A fixed LLM judge reads the whole accumulated dialogue rather than isolated replies and labels each user turn; a dialogue counts toward RCSR only if it succeeds and the mean rationality over its successful prefix is at least 0.70.

Why This Matters

Impact on research. The paper reframes a widely observed but often papered-over problem: reported gains in proactive dialogue may be artifacts of compliant simulators. By turning user resistance into a coupled training objective and supplying a metric (RCSR) that separates grounded success from sycophancy-driven success, it gives the field both a training recipe and a sharper yardstick. It also bridges two literatures that have mostly run in parallel — dialogue planning (which optimizes the system against a fixed user) and user simulation (which improves the simulator without changing how the policy is trained). The negative-leaning result is equally informative: removing a warmup component yielded the lowest average turns but halved RCSR, showing that turn-count and success-rate leaderboards can reward exactly the shortcut behavior that fails in deployment.

Real-world applications.

• Health and wellbeing coaching: motivational-interviewing-style assistants that must persist without becoming pushy, and where "user said yes too fast" is a genuine safety concern.
• Customer retention and service bots: agents that negotiate refunds, plan changes, or upsell while respecting a customer who has legitimate reasons to refuse.
• Sales and fundraising dialogue systems: the P4G charity-persuasion setting maps directly onto donation or pledge conversations where over-claiming success is costly.
• Negotiation and procurement assistants: the CraigslistBargain buyer-side results are directly relevant to automated bargaining where the counterparty actively withholds concessions.
• Evaluation and QA of conversational products: RCSR-style auditing offers a way to catch simulators or test harnesses that flatter a model's true capability.

Industry relevance. Teams shipping conversational agents typically evaluate against scripted or LLM-role-played users, and the paper quantifies how much that inflates results (e.g., P4G SR 0.859 versus 0.788 once the user is adversarial, and worst-case SR dropping to 0.737 across a simulator ensemble). The alternating self-play loop is model-agnostic and demonstrated on both Qwen3-4B and Qwen3-0.6B, making it plausible for small, deployable models — though the authors flag the added cost of two policies plus auxiliary mentalizers.

Future Directions

• Reducing dependence on synthetic supervision. The framework relies on silver BDI annotations, a teacher mentalizer, and an LLM rationality judge; the authors list bias or noise in any of these as a limitation, so learning mental states with less machine-labeled data, or making the reward robust to judge error, is an open problem.

• Validating with real users. The human audits establish agreement with synthetic components on audited samples, but the authors state explicitly that this "does not establish performance with real users" and that evaluation with legally and ethically collected real-user interactions remains future work.

• Richness of the mental-state geometry. The shared BDI representation is designed to transfer across domains but "may miss specialized user states in some tasks" — richer or task-adaptive state spaces are a natural extension.

• Cost and stability of alternating self-play. With two policies plus mentalizers, training is more expensive than single-policy planning; whether the alternation can be made cheaper, or extended beyond three tracked iterations (

Authors’ abstract

Proactive dialogue agents must guide multi-turn interactions toward task goals while adapting to users whose intentions and attitudes evolve. Existing planners improve system side action selection through search, learned policies, or user aware planning, but are often trained and evaluated against fixed or overly compliant simulators; as a result, high simulated success can reflect soft user bias rather than robust dialogue ability. We introduce MASP, a Mentalized Adversarial Self Play framework that treats proactive dialogue as a partially observable game between a system and a rationally resisting user. MASP represents the user's evolving mental state with a BDI based latent state, separates reward side teacher mentalization from policy side student mentalization, and alternates self play updates for the system and user policies. This turns robustness to rational resistance into a training objective rather than only an evaluation condition. Across diverse proactive dialogue benchmarks, compared with the widely used SFT and Planner-only approaches, MASP substantially improves success rate (SR) and rationality constrained success (RCSR), reduces average turns (AT), and performs better against hard users who make the task goal more difficult to achieve.