Flipping the Dialogue: Training and Evaluating User Language Models

AI summary

Overview

Research area: Natural Language Processing — user simulation, multi-turn dialogue evaluation, and post-training of language models.

Technical level: Advanced. The paper assumes familiarity with language model fine-tuning, perplexity, dialogue roles, and benchmark evaluation, though the core argument is accessible.

Scope: The paper argues that prompting assistant LMs to role-play users produces unrealistic simulations, and instead trains dedicated "User Language Models" that better match human conversational behavior and expose assistant weaknesses in multi-turn tasks.

What This Paper Is About

Assistant language models are post-trained to be helpful, structured, and unambiguous, while real users are terse, indirect, and refine their requests turn by turn. Prior work evaluates assistants by prompting an assistant LM to role-play a user, but the authors show these simulators are too cooperative to reflect real behavior. The goal is to train purpose-built models that generate realistic user utterances while still following a defined task intent, then use them to measure how well assistants actually perform.

Key Contributions

1. Purpose-built User Language Models (User LMs). The authors introduce UserLM-1b and UserLM-8b, trained by "flipping the dialogue" on 343,951 real human-assistant conversations from WildChat, conditioning each user turn on a high-level, generated user intent plus conversation history.

2. A demonstration that better assistants make worse user simulators. The paper reports across multiple settings that scaling or improving assistant LMs does not improve their ability to simulate users — GPT-4o is outperformed by GPT-4o-mini on some metrics, and Llama3-8b-Instruct has worse perplexity on user utterances than the smaller 1b model.

3. Six fine-grained intrinsic evaluations of simulators. First-turn diversity, intent decomposition, dialogue termination, naturalness (via the Pangram AI-detector), user role adherence, and intent adherence, with human utterances from PRISM used as a reference where possible.

4. An extrinsic evaluation on coding and math tasks. Using UserLM-8b as the simulator with GPT-4o as the fixed assistant across 650 simulations (10 per each of 65 task intents) shows assistant accuracy dropping from 74.6% to 57.4%.

Main Findings

• User LMs align better with human language. UserLM-8b achieves the lowest perplexity on both WildChat and PRISM, with 14.92 and 7.42 respectively when conditioned on intent, versus 35.02 and 20.80 for Llama3.2-1b-Instruct and 40.25 and 36.29 for Llama3-8b-Instruct. Improvements are 60-70% lower than baselines.

• Intent conditioning helps both at train and test time. All models improve with test-time intent conditioning, but the largest gains come from training with intent, which the authors argue makes the model more steerable.

• Base checkpoints beat instruction-tuned checkpoints for training users. User models initialized from base checkpoints achieve lower perplexity than those initialized from instruction-tuned ones; starting from the 1b assistant checkpoint yielded 18.45 PPL versus 27.25 from the 8b assistant checkpoint.

• User LMs produce more diverse opening turns. UserLM-8b reaches 94.55% unique 1-grams, on par with real users at 94.01% and ahead of GPT-4o at 74.42%.

• User LMs decompose intent across turns. Their average overlap with the conditioned intent is 2.69%, close to the 1.68% observed for human utterances, whereas Llama3-8b-Instruct shows 23.95% and Llama3.2-1b-Instruct 15.72%.

• Prompted assistants rarely end conversations. UserLM-8b reaches 63.54 F1 on dialogue termination versus 3-15 F1 for prompted assistant simulators.

• User LMs look more human to an AI detector. Pangram assigns an average 90.2% confidence that real user utterances are not AI-generated; prompted assistant models scored 0-3%, while User LMs scored 77-81%.

• User LMs are more robust simulators. Role adherence is 91-98% for trained user LMs versus assistant-prompted simulators that revert to the assistant role in 20-60% of conversations. Intent adherence is 93-97% for user LMs versus 54.95-88.70% for prompted assistants.

• Scaling helps User LMs but not prompted assistants. UserLM-8b outperforms UserLM-1b on all metrics, while GPT-4o outperforms GPT-4o-mini on only two of six metrics and Llama3-8b beats Llama3-1b on only one.

• Realistic simulators reveal assistant weaknesses. In simulations, GPT-4o as assistant scores 74.6 with a GPT-4o simulator, 73.2 with GPT-4o-mini, and 57.4 with UserLM-8b — roughly 17% lower.

• UserLM-8b introduces unexpected demands. It produced additional requirements not in the original intent in 43.8% of cases, mainly example test cases (34%), naming conventions (21%), and implementation constraints (20%), while GPT-based simulators added demands only 1.1-9.5% of the time.

• Pace varies more with User LMs. UserLM-8b conversations ranged from 2.1 to 6.7 turns with turn variance of 2.8, versus 3.7-5.7 turns and variance of 0.9/0.6 for the GPT-based simulators.

Methodology in Plain English

The researchers took real conversations between people and ChatGPT from the WildChat dataset, removed near-duplicates (478,498 down to 384,336), and split the data by user identity (based on hashed IP addresses and countries) at 90/5/5 so that no user appears in more than one split. This gave 343,951 training conversations, 22,442 for validation, and 17,943 for testing.

For every conversation, GPT-4o generated a "generic intent" — a high-level statement of what the user wanted, deliberately omitting specific details, guided by three hand-written examples. Then they "flipped the dialogue": instead of training a model to produce assistant responses, they formatted each conversation so the model predicts the user turns, conditioned on the intent and the conversation so far. A special <|endconversation|> token was added to the tokenizer and used as the target after the final assistant turn, so the model learns when to stop.

They fully fine-tuned Llama3-8b-Base and Llama3.2-1b-Base with a maximum sequence length of 2048 tokens, batch size 1024, learning rate 2e-5, on 4 A6000 GPUs — 62 hours for the 1b model and 227 hours for the 8b model. They compared against prompted Llama3.2-1b-Instruct, Llama3-8b-Instruct, GPT-4o, GPT-4o-mini, and the fine-tuned baseline USP-8b, engineering prompts to encourage user-like behavior and selecting the best prompt variant by validation perplexity.

Evaluation happened in two stages: intrinsic metrics on user language modeling and conversational behavior (using PRISM as an out-of-domain test set of 8,011 conversations and as a source of reference human utterances), and an extrinsic task simulation on 65 task intents based on GSM8k and HumanEval, with GPT-4o fixed as the assistant.

Why This Matters

Impact on research. The paper challenges a common assumption in multi-turn evaluation: that a stronger assistant automatically makes a better proxy for a human user. It provides released models (microsoft/UserLM-8b) and a reusable evaluation suite, giving the community a way to measure assistant performance under more realistic interaction conditions rather than overly cooperative ones.

Real-world applications:

• Multi-turn assistant benchmarking. More realistic simulators give lower, likely more honest estimates of assistant capability on coding and math tasks before deployment.
• Synthetic data generation. Because User LMs produce more varied conversations, they can generate diverse user turns for fine-tuning assistants and improving robustness.
• User modeling and survey simulation. User LMs could generate natural text responses to survey questions, extending prior work that predicts response distributions.
• Judge models. User LMs may serve as better proxies for user preferences than assistant-based judges, which carry assistant-specific biases.

Industry relevance. Any organization deploying conversational assistants has an interest in knowing how those systems behave when users are vague, iterative, or off-script. The finding that GPT-4o's measured task performance drops substantially under realistic simulation is directly relevant to product evaluation and to deciding how much headroom remains in deployed systems.

Future Directions

• Personalized User LMs. The current models simulate a broad, general audience; the authors suggest fine-tuning toward specific demographics, dialects, or domains, using the released models as a foundation to lower data requirements.
• Scaling data and model size. The authors expect that more parameters and more human-assistant conversations would yield better simulators, which in turn would support better assistant development.
• New applications beyond evaluation. User modeling, user LM judge models, and synthetic data generation are named as promising directions.
• Scoping where simulation is insufficient. The authors argue simulation is useful for discovering broad system flaws at scale, but that expert involvement — in law, creative writing, or science — remains necessary for capturing user nuance.

Target Audience

Researchers and engineers working on language model post-training, dialogue systems, and evaluation methodology, particularly those building multi-turn benchmarks or user simulators. Also useful for product teams that need realistic pre-deployment estimates of assistant performance, and for readers interested in the distinction between the user and assistant roles in conversational AI.

Authors’ abstract

Conversations with LMs involve two participants: a human user leading the conversation, and an LM assistant responding to the user's request. To satisfy this specific role, LMs are post-trained to be helpful assistants -- optimized to produce exhaustive and well-structured responses, free of ambiguity and grammar errors. User utterances, on the other hand, are rarely perfected, with each user phrasing requests in unique ways, sometimes putting in partial effort at each turn and refining on the fly. To evaluate LM performance in realistic settings, prior work simulated users in multi-turn conversations, often by prompting an LM originally trained to be a helpful assistant to act as a user. However, we show that assistant LMs make for poor user simulators, with the surprising finding that better assistants yield worse simulators. Instead, we introduce purpose-built User Language Models (User LMs) - models post-trained to simulate human users in multi-turn conversations. Through various evaluations, we show how User LMs align better with human behavior and achieve better simulation robustness than existing simulation methods. When leveraging User LMs to simulate coding and math conversations, the performance of a strong assistant (GPT-4o) drops from 74.6% to 57.4%, confirming that more realistic simulation environments lead to assistant struggles as they fail to cope with the nuances of users in multi-turn setups.