SimRPD: Optimizing Recruitment Proactive Dialogue Agents through Simulator-Based Data Evaluation and Selection

AI summary

Overview

Research area: Task-oriented dialogue systems, specifically proactive dialogue agents for recruitment, with a focus on synthetic data generation, data selection, and evaluation methodology.

Technical level: Intermediate. The paper assumes familiarity with supervised fine-tuning, reinforcement learning (PPO), and LLM-based user simulation, but its central ideas — data quality over quantity, distributional alignment, intent-transition modeling — are accessible to anyone working with LLM training pipelines.

Scope: The paper presents a three-stage framework (SimRPD) that trains a user simulator to synthesize recruitment dialogue data, evaluates and filters that data using a Chain-of-Intention protocol, and trains a deployable proactive dialogue agent on the curated subset.

What This Paper Is About

Recruitment dialogue agents must actively steer conversations with job candidates toward a concrete business outcome — persuading them to share contact information for a private-channel handoff. Training these agents requires large amounts of high-quality, goal-oriented conversation data, but such data is scarce, expensive to collect, and privacy-sensitive. The paper's goal is to solve this data bottleneck by building a simulated pipeline that generates candidate conversations synthetically, rigorously filters them for realism and distributional fit with real interactions, and trains a production agent on the survivors.

Key Contributions

1. SimRPD framework: An end-to-end closed-loop pipeline combining user simulation, multi-dimensional data evaluation, and proactive dialogue agent training, designed to overcome both data scarcity and privacy constraints in recruitment dialogue.

2. Dual-level evaluation protocol based on Chain-of-Intention (CoI): A framework that jointly measures simulator fidelity and filters synthetic dialogues using global-level metrics (KL divergence, JS divergence, Question Diversity) and instance-level metrics (Style Similarity, Result Consistency F1, Route Consistency).

3. Multi-stage training recipe: A two-stage SFT + RL pipeline for the user simulator, explicitly designed to counter LLM sycophancy and reproduce realistic candidate resistance and rejection patterns.

4. Industrial validation: A live online A/B test on a real recruitment platform with roughly 50,000 candidates, demonstrating measurable gains in contact acquisition from the deployed 8B-parameter agent.

Main Findings

• Prompt-based simulators fail distributional alignment: GPT-5.1 and Qwen3-max, used without fine-tuning, show high KL divergence from real dialogue (2.969 and 2.301 respectively) and low question diversity, indicating their intent-transition dynamics deviate sharply from real candidate behavior.

• Fine-tuning closes most of the sim-to-real gap: SFT-only simulation cuts KL divergence to 1.985; adding RL on top (SimRPD) reduces it further to 1.702 and improves JS divergence to 0.084, question diversity to 0.671, and style similarity to 0.562.

• Outcome consistency is only marginally improved: Result Consistency F1 is essentially flat across methods (0.512 for SFT-only versus 0.500 for SimRPD), and Route Consistency improves only slightly (0.08 to 0.09). Simulating whether a real candidate would actually convert remains difficult.

• Data selection beats raw data volume: Training on the unfiltered pool of 10,000 synthetic dialogues yields a 19.7% expert-annotated average error rate. Every selection method improves this, but SimRPD's dual-level filter reaches 14.2%, the best overall.

• Global and instance metrics play complementary roles: Ablating global metrics drops hallucination errors to 5.7% but worsens user experience and timing errors, raising the average to 15.0%. Ablating instance metrics raises hallucinations to 9.0% and average error to 16.4%. Instance metrics suppress hallucinations; global metrics improve conversational style and decision timing.

• Benchmarks fall short on long-horizon behavior: Even the strongest baseline (Agentic State Tracking, 16.3% average error) leaves 21.7% of errors in Timing & Logic, showing that instance-level heuristics alone do not fix intent management over multi-turn conversations.

• Real-world deployment gains: In a week-long A/B test with ~50,000 candidates, SimRPD-8B lifted contact information acquisition from 3.8% to 4.4% (+15.8% relative), while average dialogue turns rose from 4.4 to 6.0 (+36.4%).

• Intent classification is reliable: The underlying intent classifier achieves 97.6% accuracy against 1,000 human-annotated utterances, making the CoI-based metrics dependable as a foundation.

Methodology in Plain English

The authors model a recruitment conversation as a sequence of discrete user intentions — for example, an information inquiry followed by positive intent, followed by successful conversion. Each turn is labeled with one of nine intention categories, and the whole conversation becomes a "chain of intention."

Step 1: Build a user simulator. Starting from anonymized real candidate profiles (gender, age, work experience, job preferences), they fine-tune a Qwen3-8B model on real (profile, context, response) triplets so it learns how job seekers actually talk. They then apply reinforcement learning with penalties for repetition, unnatural length, and inconsistent actions, which pushes the simulator to reproduce realistic rejection and skepticism rather than agreeable compliance.

Step 2: Generate and filter data. The simulator converses with the agent to produce 10,000 dialogues. These are then evaluated two ways. At the global level, the authors compare the synthetic conversation dynamics against real data by building a transition matrix over intentions — essentially a heatmap of "given that the current turn is intent X, what was the previous turn?" — and measuring KL and JS divergence between the synthetic and real matrices. They also measure question diversity via Shannon entropy over clustered question embeddings. At the instance level, an LLM judge scores stylistic resemblance to real dialogues, a graph check verifies that each intent path is structurally valid against historical data, and an F1 score checks whether the simulator's conversion decisions match real ones.

Step 3: Select the best subset. They retain the top 1,000 of 10,000 dialogues using two strategies. For per-dialogue quality, they rank by composite instance scores. For dataset-level distribution match, they use Monte Carlo sampling — repeatedly drawing random 1,000-dialogue subsets and keeping the one with the lowest divergence from real data — and greedy backward elimination, which iteratively removes the dialogue whose absence most reduces distributional error.

Step 4: Train the agent. The proactive dialogue agent is trained on the curated subset, first by supervised fine-tuning on single-turn context-response pairs, then with PPO. The reward combines a rule-based safety term (penalizing toxic language, repeated questions, privacy violations) and a model-based preference term from a reward model trained on human expert comparisons.

Why This Matters

Impact on research: The paper moves the synthetic-data conversation beyond "generate more" toward "generate and measure carefully." Its CoI framework offers a concrete, domain-generalizable way to evaluate whether simulated users behave like real ones at both the population and individual level — a question most simulator papers address only with fluency or coherence scores. The demonstrated trade-off between instance-level and global-level filtering criteria is a useful design insight for anyone building data-curation pipelines.

Real-world applications:

• Recruitment and job matching platforms that need to engage passive candidates at scale without exposing real users to untested conversational strategies.
• Other high-stakes persuasive dialogue domains where a measurable handoff or conversion event defines success, such as insurance, real estate, and financial product advisory.
• Customer service and lead qualification, where agents must decide when to escalate from chat to a human or to a contact exchange.
• Any privacy-constrained setting where real interaction logs cannot be used directly for training but a reference set exists for distributional comparison.

Industry relevance: The results are reported from a live deployment on commercial traffic, not a benchmark. A 15.8% relative improvement in contact acquisition over one week on 50,000 candidates is the kind of number that drives production decisions. The paper also quantifies a cost — longer conversations — which is exactly the kind of trade-off operations teams need to see before adopting a system. The authors include a research team from industry and academia, and the training configuration (Qwen3-8B backbone, 8x H20 GPUs) reflects realistic deployment constraints rather than frontier-scale compute.

Future Directions

1. Turn evaluation metrics into reward signals: The authors propose using the CoI-based evaluation metrics directly as rewards so that user simulators can self-evolve without a separate data-filtering stage, potentially enabling continuous improvement across broader negotiation domains.

2. Reduce domain engineering cost: The framework is not plug-and-play — profile schemas and the intention graph are hand-tailored to recruitment. Making the intention taxonomy transferable across domains would substantially widen applicability.

3. Model irrational human behavior: Current LLM simulators lean toward logical, coherent responses and struggle with sarcasm, emotional volatility, and unpredictable shifts. Closing this gap is necessary before agents can be robust against long-tail real-user behavior.

4. Address the cold-start problem: Global distributional metrics depend on a high-quality reference set of real dialogues. For new markets or new job categories with no historical data, only instance-level evaluation is currently possible, which the ablation results show is insufficient on its own.

Target Audience

Machine learning engineers and applied researchers building task-oriented or proactive dialogue agents for commercial deployment; data-centric AI practitioners interested in synthetic data generation and subset selection strategies; and product or growth teams at recruitment, marketplace, or lead-generation platforms evaluating whether simulator-driven training pipelines are worth the engineering investment. Readers focused purely on open-domain chat or general LLM pretraining will find the domain-specific intent taxonomy less directly applicable, though the dual-level evaluation framing transfers well.

Authors’ abstract

Task-oriented proactive dialogue agents play a pivotal role in recruitment, particularly for steering conversations towards specific business outcomes, such as acquiring social-media contacts for private-channel conversion. Although supervised fine-tuning and reinforcement learning have proven effective for training such agents, their performance is heavily constrained by the scarcity of high-quality, goal-oriented domain-specific training data. To address this challenge, we propose SimRPD, a three-stage framework for training recruitment proactive dialogue agents. First, we develop a high-fidelity user simulator to synthesize large-scale conversational data through multi-turn online dialogue. Then we introduce a multi-dimensional evaluation framework based on Chain-of-Intention (CoI) to comprehensively assess the simulator and effectively select high-quality data, incorporating both global-level and instance-level metrics. Finally, we train the recruitment proactive dialogue agent on the selected dataset. Experiments in a real-world recruitment scenario demonstrate that SimRPD outperforms existing simulator-based data selection strategies, highlighting its practical value for industrial deployment and its potential applicability to other business-oriented dialogue scenarios.