MADS: Multi-Agent Dialog Simulation for Robust Conversational Agents

AI summary

Overview

Research area: Natural language processing, specifically task-oriented dialogue, function calling and tool use by LLM agents, synthetic data generation, and reinforcement learning post-training.

Technical level: Intermediate. The paper assumes familiarity with LLM post-training (supervised fine-tuning, RL with verifiable rewards, GRPO) and with existing function-calling benchmarks, but the pipeline and metrics are described concretely enough for a reader with general machine-learning background.

One-sentence scope: The paper introduces MADS, a three-agent LLM simulation pipeline that generates validated multi-turn tool-calling dialogues with deliberately injected user and backend adversities, and uses it to build MADS-Bench, a six-domain benchmark on which Qwen-2.5 7B and 32B models are trained with SFT followed by RLVR.

What This Paper Is About

Deployed task-oriented conversational agents are brittle: they are usually trained and evaluated on cooperative users who supply complete information and whose requests are satisfiable, so they fail when users give partial details, change goals, express frustration, or ask for things that are impossible. Existing function-calling benchmarks (BFCL, ComplexFuncBench, ToolBench, τ-bench, τ²-bench) largely evaluate cooperative behavior and do not systematically expose agents to non-cooperative dynamics, tool failures, or infeasible requests. The paper's goal is to close that resource gap by generating structured, validated multi-turn dialogue trajectories that contain both "unhappy path" and "impossible" conversations, and then measuring whether training on them makes agents more robust.

Key Contributions

1. MADS, a multi-agent dialog simulation pipeline that generates structured multi-turn dialogues with explicit reasoning (<think>), planning (<plan>), and tool invocation (<action>), and that allows systematic injection of conversational and backend adversities at varying levels of scenario complexity.
2. MADS-Bench, a synthetic multi-turn function-calling benchmark spanning normal, unhappy, and impossible scenarios across multiple domains and incorporating Pre-Act-style reasoning and planning, intended to be efficient across different post-training methods.
3. A taxonomy that decomposes task-oriented dialogue complexity into conversational adversities (partial information, incremental constraint revelation, goal revision, adversarial tone) and system execution adversities (backend failure, empty results, timeout, policy constraint), plus an impossible flag with a specified fallback_behavior.
4. Controlled empirical evaluations using Qwen-2.5 models (7B and 32B), comparing training on MADS-Bench with training on APIGen, and comparing supervised fine-tuning (SFT) alone against SFT followed by reinforcement learning with verifiable rewards (RLVR).

Main Findings

• MADS supervision outperforms APIGen supervision at the turn level. For Qwen2.5-7B under Pre-Act prompting, SFT on MADS reached 0.60 action recall, 0.70 tool F1, 0.44 parameter match, 0.25 response F1, and 0.51 response similarity, versus 0.32, 0.50, 0.38, 0.1, and 0.15 for SFT on the converted APIGen data, and 0.22, 0.38, 0.38, 0.07, and 0.14 for the base model.
• RLVR improves structured grounding metrics. On MADS-trained models, Qwen2.5-7B with ReAct prompting improved from 0.54 to 0.67 action recall and 0.54 to 0.80 tool F1 after RLVR; Qwen2.5-32B with Pre-Act improved from 0.70 to 0.82 tool F1 and 0.66 to 0.70 action recall. The paper states RLVR consistently improves tool-name selection and parameter matching more reliably than semantic answer quality.
• Semantic answer metrics do not improve uniformly. For Qwen2.5-7B with Pre-Act, response F1 moved from 0.25 to 0.21 and response similarity from 0.51 to 0.46 after RLVR, while the 32B Pre-Act model stayed at 0.32 response F1 and 0.57 response similarity.
• Fewer hallucinated actions after RLVR, but the paper frames SFT as insufficient. SFT improved format adherence and tool-use frequency but left the assistant hallucinating task completion without tool calls, generating ungrounded tool arguments (nonexistent IDs, incorrect temporal values), and contradicting tool outputs such as promising a refund when the tool returned a zero-refund result.
• Trajectory-level (L-2) success rises with RLVR. Qwen2.5-7B went from 24.49% to 39.78% task success and Qwen2.5-32B from 44.08% to 53.96%. The paper notes the smaller relative gain for the 32B model suggests it internalizes structured supervision more effectively during SFT, leaving less headroom for RL refinement.
• Prompting strategy interacts with supervision and model size. Pre-Act prompting generally yielded higher action recall under MADS supervision, particularly for smaller models, while ReAct showed competitive or stronger tool F1 in some settings (for example, Qwen-7B SFT+RLVR).
• Gains on unhappy paths are large; impossible scenarios degrade under RLVR. By scenario complexity, MADS SFT versus MADS SFT+RLVR moved from 23.25% to 39.04% (7B) and 68.86% (32B) on unhappy paths, compared with 29.04% for the APIGen SFT 7B baseline, and from 23.57% to 40.28% (7B) and 48.94% (32B) on normal scenarios, compared with 35.82% for APIGen. On impossible scenarios, RLVR reduced performance (7B: 40.00% to 36.67%; 32B: 63.33% to 56.67%), which the paper attributes to only five impossible instances in the limited RL fine-tuning set.
• APIGen occasionally wins under SFT alone. The paper reports that APIGen-trained models occasionally achieve higher trajectory success than MADS-trained models under supervised fine-tuning, because APIGen contains primarily cooperative trajectories with deterministic tool execution patterns that encourage stable but shallow decision policies. Home Services, held out from all training, showed 40.92% for APIGen SFT, 28.87% for MADS SFT 7B, 49.95% for MADS SFT 32B, 42.50% for MADS SFT+RLVR 7B, and 53.87% for MADS SFT+RLVR 32B, following the column order given in the table.
• Larger models handle hard conversations better. The paper reports that the 32B model performs better overall than the 7B version, with the advantage especially large for unhappy and impossible conversations.
• Dataset composition figures. The dataset statistics table reports 574 training turns across 106 conversations and 46 scenarios (average 5.42 turns), and 521 test turns across 83 conversations and 54 scenarios (average 6.28 turns), totaling 1095 turns, 189 conversations, and 87 scenarios. The abstract describes MADS-Bench as spanning six domains, 55 personas, and 1095 conversations, with approximately 66% of conversations containing failure-prone scenarios. Within the splits, the paper reports that Banking dominates the training set (62.3%), unhappy-path scenarios account for 42.5% of training conversations, and the test set is predominantly normal scenarios (71.1%) with unhappy-path and impossible scenarios making up the remaining 28.9%.

Methodology in Plain English

The authors built a simulator in which three LLM agents talk to each other. An assistant agent plays the system being trained or tested: at each step it sees the conversation history and the available tool definitions, and it emits a reasoning block, an optional plan, and one action, where the action is either a tool call or something said to the user. A user agent plays the human, initialized with a persona, task objective, known slot values, and any injected behaviors, and it ends the exchange with an explicit [DONE] token when the task is finished or abandoned. A tool agent plays the backend, given tool definitions plus scenario seed data, and it can return realistic edge cases such as time-outs and empty results. Each agent sees only its own view of the history: the assistant sees everything including tool calls and results, the user sees only user-facing exchanges, and the tool agent sees only tool invocations and their context. A central runner manages turn-taking and termination.

Scenarios are defined as JSON configurations containing a task specification (objective, slot values split into hard constraints and soft preferences, and success criteria), domain-specific tool definitions (72 unique tools across all domains), and seed data representing realistic backend state. Six domains are covered: banking assistance, online shopping, calendar assistance, restaurant booking, travel assistance, and home services. Fifty-five personas derived from the PersonaHub dataset, each with a name, age, hometown, occupation, biographical background, writing style description, and sample messages, are assigned to scenarios by domain relevance so that the user agent's phrasing stays stylistically consistent.

Adversities are injected in two families: conversational adversities that stress planning and clarification (partial information, incremental constraint revelation, goal revision, adversarial tone) and execution-level adversities that require recovery or principled refusal (backend failure, empty results, timeout, policy constraint). Scenarios whose objective is literally unattainable are flagged impossible with a fallback_behavior describing how the assistant should decline and offer alternatives.

Generated traces pass through a four-check automatic validation combining rule-based checks and LLM-as-judge evaluators: a syntax evaluator (structural correctness, including that the first step of each turn contains a plan, each turn ends with exactly one say action, and tool calls conform to schema), a faithfulness evaluator (detecting hallucinated parameter values and utterances unsupported by tool results), a role confusion evaluator (filtering conversations where the user agent started helping the assistant), and a task success evaluator that respects the distinction between hard constraints and soft preferences and, for impossible scenarios, requires appropriate explanations and alternatives.

Evaluation happens at two levels. L-1 turn-level evaluation measures action classification recall, tool call accuracy (right tool, right parameters, all required parameters), and conversation quality as cosine similarity between predicted and reference utterances using sentence-transformers/all-mpnet-base-v2. L-2 trajectory-level evaluation scores whole conversations with a stop-on-first-error rule, where a conversation's score is the fraction of correct turns up to the first incorrect turn (correct/attempted) and the reported success rate is the mean conversation score.

Training is a two-stage pipeline. SFT trains on the full MADS-Bench training split of 106 conversation trajectories with learning rate 2×10⁻⁴, batch size 4, and gradient accumulation over 4 steps, predicting the structured output tokens. RLVR then runs on a subset of 50 sampled training conversations with learning rate 1×10⁻⁵, batch size 1, gradient accumulation of 4 steps, for 5 epochs, generating K=8 candidate responses per rollout and optimizing with the Group Relative Policy Optimization (GRPO) objective. Rewards are fully deterministic and computed against gold reference actions rather than live execution outcomes: an action-type reward (tool vs. say), a tool-accuracy reward computed only when both gold and predicted actions are tool calls (tool name match plus exact parameter dictionary match), and a semantic reward computed as the cosine similarity of sentence-transformer embeddings between predicted and gold utterances, defaulting to 0.0 if either action is not a say action. The total reward is the sum, bounded in [0, 3]. For comparison, 198 APIGen-MT conversations were converted into the same format with human, gpt, function_call, and observation turns remapped and <think>/<plan> blocks synthesized by GPT-4o-mini, then passed through the same validation pipeline to yield 750 training samples; Qwen2.5-7B-Instruct was fine-tuned on those with the same QLoRA configuration.

Why This Matters

The paper targets a widely reported failure mode of enterprise conversational AI: systems that behave acceptably when users cooperate but break down under incomplete information, changing goals, or unsatisfiable requests. By making adversarial trajectories a first-class training and evaluation resource, it offers a recipe for measuring and improving robustness rather than only task accuracy on cooperative data. It also contributes a concrete taxonomy that other researchers can reuse to categorize failure modes.

Real-world applications:

• Customer support and enterprise service workflows, where users often omit order numbers or confirmation codes and agents must ask for the right missing information before acting.
• Banking and transaction handling, where policy constraints such as an expired refund window require the agent to refuse gracefully and offer alternatives.
• Scheduling and booking systems (calendar, restaurant, travel), where users revise constraints mid-conversation and where tools can return empty results or fail outright.
• Home services dispatch, which functions in this work as the held-out domain for measuring cross-domain generalization.

Industry relevance:

Authors’ abstract

Task-oriented conversational agents remain fragile under realistic conditions, especially when users exhibit non-cooperative behavior. Existing function-calling benchmarks underrepresent adversarial conversation trajectories, thereby limiting the training resources available for developing robust agents. We present MADS (Multi-Agent Dialog Simulation), a synthetic data generation pipeline that produces structured, diverse dialogue trajectories. We inject adversarial conversation, producing validated conversation trajectories that capture realistic, unhappy paths. Using MADS, we construct MADS-Bench: a benchmark spanning six domains, 55 personas, and 1095 conversations, with approximately 66% of conversations containing failure-prone conversation scenarios. We evaluate the robustness of the trained models using a two-stage training approach that combines supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR).