Agentic Persona Control and Task State Tracking for Realistic User Simulation in Interactive Scenarios

AI summary

Overview

• Research area: Human-Computer Interaction, specifically multi-agent LLM systems for simulating human users in goal-oriented conversations.
• Technical level: Intermediate. Familiarity with LLM agents, tool calling, and dialogue state tracking helps, but the architecture is described in plain, component-by-component terms.
• Scope: The paper proposes and ablates a three-agent framework (User Agent, State Tracking Agent, Message Attributes Generation Agent) for simulating restaurant guests, evaluated on 60 ordering test cases across five metrics.

What This Paper Is About

Testing conversational AI at scale requires realistic user behavior, but static test sets cannot capture multi-turn dynamics and human evaluators are expensive and hard to standardize. Single-LLM user simulators tend to overload one model with state tracking, behavior modeling, response generation, and context management at once, producing either scripted or erratic interactions. The paper's goal is to decompose user simulation into specialized cooperating agents that separately track task progress and control persona-driven behavioral attributes, then measure whether that decomposition produces more realistic, controllable, and explainable simulations.

Key Contributions

1. A multi-agent framework for human user simulation with specialized agents, intended to improve realism, controllability, and explainability through persona control and task state grounding.
2. A systematic evaluation methodology built on ablation studies and standardized metrics for persona adherence, task completion accuracy, decision explainability, and overall simulation quality.
3. A test dataset in the restaurant ordering domain containing 60 ordering test cases paired with 20 personas and a menu of 50+ items, used to validate the framework in multi-turn scenarios.

Main Findings

• Full system outperforms the baseline on every metric: Config5 (full multi-agent system) scored PAS 0.706, BVS 0.839, TRA 0.785, DEI 0.994, and CRRS 0.818, versus baseline Config1 at PAS 0.589, BVS 0.218, TRA 0.608, DEI 0.000, and CRRS 0.404.
• Statistically significant gains: Reported improvements from Config5 over baseline were PAS +19.9% (p = 0.0037), BVS +284.5% (p = 0.0000), TRA +29.1% (p = 0.0047), DEI +100.0% (p = 0.0000), and CRRS +102.6% (p = 0.0000).
• The Message Attributes agent drives persona adherence on its own: Config4 (User Agent + Message Attributes Generation Agent) achieved the highest individual PAS at 0.661, yet its BVS was 0.000, which the paper interprets as over-rigid behavioral control without order-state awareness.
• The State Tracking agent drives task accuracy: Config3 (User Agent + State Tracking Agent) and Config5 both reached TRA 0.785, the highest task restriction adherence values observed, confirming the State Tracking Agent's role in order accuracy.
• Component synergy: Neither sub-agent alone matched the full system; Config3 reached CRRS 0.651 and Config4 reached 0.462, both below Config5's 0.818.
• Explainability requires the full stack: Only Config5 approached near-perfect DEI (0.994); Config1 had DEI 0.000, Config2 had 0.200, Config4 had 0.432, and Config3 had 0.498.
• Cost profile across configurations: Average tokens and latency per response were 6,618 / 5.08s (Config1), 13,505 / 4.56s (Config2), 24,580 / 36.30s (Config3), 15,763 / 16.88s (Config4), and 14,789 / 23.16s (Config5). The paper describes the full system as balancing cost at 14,789 tokens with the best performance.
• Acknowledged overhead: The limitations section states the architecture incurs 124% more tokens and 356% higher latency than the baseline.

Methodology in Plain English

The framework splits the job of pretending to be a restaurant customer across three cooperating AI agents built with Pydantic AI and GPT-4o.

The User Agent is the orchestrator. It receives the incoming message from the ordering system and writes the simulated guest's reply, calling tools to fetch the persona biography and to invoke the other two agents when needed. The State Tracking Agent keeps two lists: T_current, the items confirmed so far in the order, and T_target, the final order the guest is trying to reach. It uses tools to add, remove, or clear items and enforces that state updates are monotonic. The Message Attributes Generation Agent reads the persona biography and current task state and outputs four behavioral attributes for the next message: mood tone (casual, frustrated, confused, enthusiastic), task execution style (one-by-one or all-at-once), exploration style (explores or does-not-explore), and task completion status (complete or incomplete).

Structured constraints govern the agents: the User Agent must call sub-agents in a fixed order (State Tracking then Message Attributes Generation), state updates must stay within the target bounds, and behavioral attributes must stay within persona-appropriate ranges. The conversation follows rules from an initial greeting through progressive state building and clarification handling to closure. The simulation exits when the Message Attributes Generation Agent marks the task complete, defined as T_current being a superset of T_target.

Evaluation used 20 personas each paired with 3 target orders of increasing complexity (simple, medium, complex), giving 60 test cases. The simulated guest conversed with an independent GPT-4o ordering system that only saw conversation history and had no access to test case data or the guest's internal state. Five ablation configurations were run: a single LLM baseline with all information in the prompt, User Agent only, User Agent + State Tracking, User Agent + Message Attributes Generation, and the full system. Five metrics were computed: Persona Adherence Score (PAS) as a weighted average of four binary component matches, Behavioral Variance Score (BVS) from average behavioral transition rates peaking at a 20% transition rate, Task Restriction Adherence (TRA) as an F1 score over normalized task items, Decision Explainability Index (DEI) based on counts of explained decisions from tool invocations, and a Composite Realism and Reliability Score (CRRS) = 0.25·PAS + 0.20·BVS + 0.35·TRA + 0.20·DEI. Experiments ran on a local machine with a 10-core Apple M1 Max CPU, 32GB of RAM, and 3.2GHz clock speed.

Why This Matters

Impact on research: The paper argues it is the first to explore explainable realistic human user simulation through a multi-agent architecture combining dedicated agentic task tracking with fine-grained message generation attribute control. Its evaluation strategy deliberately uses ablation studies rather than head-to-head comparison with existing user simulation approaches, which the authors say operate under fundamentally different architectural assumptions. It also backs up prior related work claims that combining LLMs with additional components better captures cognitive processes.

Real-world applications:

• Automated regression testing of restaurant ordering assistants against diverse, persona-driven guest behavior.
• Customer support systems, where the paper lists cross-domain validation as a future direction.
• E-commerce ordering and healthcare consultation systems, both explicitly named as candidate domains.
• Educational tutoring and financial advisory or travel booking agents, also listed as target domains for extending the framework.

Industry relevance: The work is authored at Toast Inc. and funded by Toast Inc., with acknowledgements to a Principal Data Scientist and a Director of AI at the company, indicating direct commercial interest in scalable conversational AI testing. The framework promises systematic, high-quality interaction data generation for testing, evaluation, and quality assurance wherever validating human-system interactions matters. The stated limitations matter for deployment decisions: 124% more tokens and 356% higher latency than the baseline, plus English-only, single-domain validation over 60 test cases.

Future Directions

• Adaptive persona evolution: enabling dynamic behavioral adjustment during conversations rather than fixed persona parameters.
• Multi-modal integration: adding voice-based prosodic features, visual gesture recognition, and emotional sentiment tracking to the simulation.
• Cross-domain validation: testing the architecture in customer support, e-commerce ordering, healthcare consultations, educational tutoring, financial advisory, travel booking, and technical troubleshooting.
• Efficiency optimization: agent caching, selective tool invocation, and smaller specialized models to address the computational overhead the paper reports.
• Open modeling gaps: the limitations note that behavioral modeling lacks complex human behaviors such as indecision, social dynamics, and cultural nuances, and that validation is English-only and single-domain.

Target Audience

Researchers and engineers working on conversational AI evaluation, LLM agent orchestration, or dialogue state tracking will get the most from this paper, since its central contribution is an architectural decomposition plus an ablation methodology. Product and QA teams at companies deploying customer-facing assistants will find the practical test-case design and cost metrics relevant. Readers interested in persona modeling and synthetic data generation will find the behavioral attribute schema and the documented failure mode of rigid behavioral control (BVS = 0 in Config4) directly useful. Beginners can follow the architecture descriptions, but the metric definitions and ablation reasoning assume some comfort with F1 scores, statistical significance testing, and agent tool-calling patterns.

Authors’ abstract

Testing conversational AI systems at scale across diverse domains necessitates realistic and diverse user interactions capturing a wide array of behavioral patterns. We present a novel multi-agent framework for realistic, explainable human user simulation in interactive scenarios, using persona control and task state tracking to mirror human cognitive processes during goal-oriented conversations. Our system employs three specialized AI agents: (1) a User Agent to orchestrate the overall interaction, (2) a State Tracking Agent to maintain structured task state, and (3) a Message Attributes Generation Agent that controls conversational attributes based on task progress and assigned persona. To validate our approach, we implement and evaluate the framework for guest ordering at a restaurant with scenarios rich in task complexity, behavioral diversity, and conversational ambiguity. Through systematic ablations, we evaluate the contributory efficacy of each agentic component to overall simulation quality in terms of persona adherence, task completion accuracy, explainability, and realism. Our experiments demonstrate that the complete multi-agent system achieves superior simulation quality compared to single-LLM baselines, with significant gains across all evaluation metrics. This framework establishes a powerful environment for orchestrating agents to simulate human users with cognitive plausibility, decomposing the simulation into specialized sub-agents that reflect distinct aspects of human thought processes applicable across interactive domains.