MirrorBench: A Benchmark to Evaluate Conversational User-Proxy Agents for Human-Likeness

AI summary

Overview

Research area: Evaluation methodology for large language model (LLM) agents that simulate human users in conversation — a topic at the intersection of dialogue systems, LLM-as-a-judge evaluation, and computational linguistics.

Technical level: Intermediate. The paper assumes familiarity with LLM prompting, lexical-diversity statistics, and standard evaluation concepts such as confidence intervals and correlation coefficients, but the core ideas are explained clearly enough for readers who know conversational AI basics.

Scope (one sentence): The paper introduces MirrorBench, an open-source framework that measures how human-like a "user-proxy" LLM's utterances are, using lexical-diversity metrics and calibrated LLM-judge metrics across four conversational datasets and six proxy models.

What This Paper Is About

Practitioners increasingly prompt LLMs to "act as a user" so they can generate synthetic conversations for testing assistants and for fine-tuning data, but naive prompting tends to produce verbose, overly cooperative utterances that do not resemble real people. The paper's goal is to measure the human-likeness of these user-proxy agents rigorously and independently of whether the downstream assistant succeeds at a task. To do this, the authors build MirrorBench, which compares proxy-generated user turns against real human user turns under matched conversational conditions.

Key Contributions

1. A benchmark protocol for evaluating user-proxy agents on human-likeness, using human-anchored normalization and calibrated judge metrics, with explicit separation from downstream task success.

2. Six metrics in two families: lexical-diversity metrics (MATTR, HD-D, Yule's K) and judge-based realism metrics (GTEval, Pairwise Indistinguishability, Rubric-and-Reason), contextualized by Human–Human (HH) and Proxy–Proxy (PP) calibration controls.

3. A unified preprocessing of four open conversational datasets — QULAC, ClariQ, OASST1, and ChatbotArena — for user-proxy benchmarking, yielding 795 conversations with real human user turns.

4. An empirical study across six user-proxy LLMs (GPT-4o, GPT-5, GPT-OSS-120B, Claude-4-Sonnet, Gemini-2.5-Pro, and a LLaMA-based user-simulation proxy, USP), reporting realism–diversity tensions, judge sensitivity, human-judge correlation, and assistant sensitivity analysis. The framework is open source with a command-line interface.

Main Findings

• Judge-based realism favors Gemini-2.5-Pro and Claude-4-Sonnet. Across GTEval, PI Δw, and RNR, these two models are the most human-like on every dataset, with GPT-4o competitive but generally behind, and GPT-OSS-120B and GPT-5 trailing. On ClariQ and QULAC, Claude-4-Sonnet and Gemini-2.5-Pro approach the HH ceiling under RNR.

• Lexical diversity diverges from the realism ranking. On ClariQ, Claude-4-Sonnet and GPT-5 exceed the human anchor on MATTR/HD-D and exhibit lower Yule's K, meaning a more diverse vocabulary than human questioners in that information-seeking regime. QULAC shows a uniform diversity deficit: every proxy falls below the human baseline on MATTR and HD-D, with positive shifts in Yule's K, indicating more templated clarifications than humans. ChatbotArena and OASST1 sit between these extremes.

• A realism–diversity tension exists. Proxies that lead on judge realism (Claude-4-Sonnet, Gemini-2.5-Pro) can still under-shoot diversity on QULAC, suggesting judges favor intent and style over surface variety. Gemini-2.5-Pro and GPT-4o give the strongest diversity alignment with humans across datasets, while Claude-4-Sonnet generally overshoots the human baseline; GPT-5 and GPT-OSS-120B show larger diversity differences from the human baselines.

• Judge choice shifts both absolute scores and orderings. With assistant and user-proxy both fixed to GPT-4o on ChatbotArena, GTEval scores spread from approximately 0.45 to 0.81, with GPT-4o as judge giving the highest value. PI is the most volatile: Gemini-2.5-Pro and Claude-4-Sonnet produce near-zero or negative win deltas, while GPT-5 and GPT-4o are clearly positive. RNR saturates near the ceiling for GPT-4o and GPT-5 judges (approximately 0.96–0.98) and is lower for others (approximately 0.79–0.82), giving a sensitivity ordering of PI > GTEval > RNR. Claude-4-Sonnet behaved as a conservative yet stable judge, which is why it was used for the primary results.

• Judge scores track human perception, with PI weaker than GTEval. On ChatbotArena, Claude-4-Sonnet judge scores were correlated with blinded human expert annotations on 100 stratified episodes per metric (GTEval, PI), using Spearman's ρ, Pearson's r, and Kendall's τ (all p < 0.001). GTEval aligned strongly with human judgments; PI showed moderate correlation, which the authors attribute to the inherent difficulty of annotating pairwise comparisons. A broader study spanning three datasets and three proxy models (50–100 human-annotated episodes per condition) produced GTEval ρ from 0.607 to 0.697 and PI ρ from 0.532 to 0.671, all p < 0.001.

• A small specialized proxy competes with much larger general models. USP, a LLaMA-based model fine-tuned for user simulation, outperformed several much larger general-purpose LLMs on GTEval and RNR across all four datasets and on PI across two datasets. It produced richer vocabulary than real users on ChatbotArena, OASST1, and ClariQ, but its main limitation surfaced on QULAC, where PI deviation was notably high.

• Confidence intervals are generally narrow, with exceptions. Lexical-diversity intervals occasionally widen, notably for Yule's K, because very short user-side text (few turns) increases estimator variance. Where judge-metric intervals overlap (for example, GTEval on ChatbotArena), rank differences are small and agree with PI/RNR trends.

• Rollout drift is rare. The assistant is given the reference chat history to anchor rollouts to the same trajectory; manual inspection found divergence in fewer than 1% of samples due to LLM hallucination.

• Assistant sensitivity and rank stability. The paper reports an analysis on ChatbotArena holding the judge fixed and varying the assistant among GPT-4o, Claude-4-Sonnet, and Gemini-2.5-Pro, with per-metric proxy ranks computed from the scores. The specific numerical results of this analysis are not included in the provided text, which is truncated at that point.

Methodology in Plain English

The researchers start from four public conversation datasets that contain real human user turns. To make the comparison fair, they normalize every conversation into alternating user–assistant turns, keep only English dialogues with at least two turns, and use stratified sampling with a fixed random seed to select up to 200 samples per dataset, ensuring coverage of different conversation lengths, topics, and clarification patterns.

For each real conversation, an auxiliary LLM writes a goal description summarizing the user's intent, behavior, tone, and persona. That description becomes the initialization prompt for the user-proxy model under test.

The proxy then replays each conversation against a fixed assistant model. Crucially, the assistant is conditioned on the original reference dialogue so the synthetic rollout follows the same conversational path as the real one — this means the proxy and the real human are compared under identical context. Only the proxy's user turns are scored; assistant output is ignored.

Two families of metrics are then applied. The lexical-diversity group (MATTR with window size 50, Yule's K, and HD-D with sample size 42) measures vocabulary richness and repetition. Because raw values depend on sequence length and domain, proxy scores are converted to z-scores relative to the distribution of real human user text on the same dataset, so a z-score near 0 means human-level behavior. The judge group (GTEval, Pairwise Indistinguishability, and Rubric-and-Reason) uses an LLM to assess behavioral realism such as tone and naturalness, with chain-of-thought prompting and repeated judgments (c = 1 for GTEval, c = 3 for PI, c = 2 for RNR) aggregated for stability.

To make judge scores interpretable, the authors add two controls: Human–Human, where a human conversation is compared to itself (approximating the judge's ceiling for genuine human behavior), and Proxy–Proxy, where a proxy conversation is compared to itself (capturing the judge's score for the proxy in isolation). For PI, these anchors are used in an affine rescaling to produce a calibrated [0,1] score. Results are aggregated with means and two-sided 95% confidence intervals from the Student-t distribution, and robustness checks swap assistants, swap judges, and repeat runs across seeds.

Why This Matters

Impact on research. The paper reframes user simulators from tools used to evaluate other systems into objects of study in their own right. By showing that judge realism and lexical diversity can point in different directions, it argues against relying on a single metric family, and it provides calibration controls (HH/PP) and human-correlation evidence that make judge-based comparisons more trustworthy. It also gives the community a reproducible, open-source harness with preprocessing for four public datasets.

Real-world applications:

• Regression testing for assistants. Teams that use simulated users to check whether an assistant update broke something can now measure whether the simulated users themselves are realistic, so failures are not artifacts of an unrealistic simulator.

• Synthetic data generation for fine-tuning. When user-proxy outputs are used as post-training data, unrealistic verbosity or over-cooperation introduces distribution shift; human-likeness metrics help filter or select proxies before generating large volumes of data.

• Stress testing without human participants. Scenarios that are infeasible or unsafe to test with real users can be approximated by proxies, but only if those proxies behave plausibly.

• Model and vendor selection. Organizations choosing which LLM to use as a simulated user can compare candidates on human-likeness across multiple conversational regimes rather than relying on leaderboards for assistant tasks.

Industry relevance. The work comes from SAP Labs and targets practical pipelines: it includes cost, latency, and throughput considerations, and the authors emphasize that a small fine-tuned open-source proxy (USP) can rival much larger proprietary models on realism, which matters for teams weighing deployment cost against quality.

Future Directions

• Closing the clarification-centric gap. QULAC showed a uniform diversity deficit and was the dataset where the specialized proxy's PI deviation was notably high, so improving proxy behavior in clarification-heavy, task-oriented dialogue is an open problem.

• Better lexical calibration. Several proxies either overshoot or undershoot human diversity depending on the dataset, and Yule's K had wide confidence intervals on short user text; better estimators or length-robust normalization could stabilize these measurements.

• Reducing judge sensitivity. Absolute scores and even fine-grained orderings shifted with the judge model. The authors recommend multi-judge evaluation and HH/PP calibration, but a principled way to normalize across judges or aggregate them remains unresolved.

• Extending to more regimes and proxies. The benchmark covers four datasets and six proxies; whether the realism–diversity tension generalizes to other domains, languages, and proxy architectures is not established here.

Target Audience

This paper benefits researchers and engineers who build or use conversational user simulators, teams that generate synthetic dialogue data for post-training, and evaluation specialists who work with LLM-as-a-judge methods and need to understand their biases. It is also relevant to practitioners selecting models for assistant testing pipelines, and to dialogue-systems researchers interested in how lexical statistics and judged realism relate to each other.

Authors’ abstract

Large language models (LLMs) are increasingly used as human simulators, both for evaluating conversational systems and for generating fine-tuning data. However, naive "act-as-a-user" prompting often yields verbose, unrealistic utterances, motivating principled evaluation of *user proxy agents*. We present **MirrorBench**, a reproducible and extensible benchmarking framework that evaluates user proxies solely on their ability to produce human-like user utterances across diverse conversational regimes, explicitly decoupled from downstream task success. **MirrorBench** combines three lexical-diversity metrics (**MATTR**, **Yule's~$K$**, and **HD-D**) with three LLM-judge-based metrics (**GTEval**, **Pairwise Indistinguishability**, and **Rubric-and-Reason**), and contextualizes judge scores using Human-Human and Proxy-Proxy calibration controls. Across four public datasets, **MirrorBench** yields variance-aware comparisons and reveals systematic gaps between user proxies and real human users. The framework is open sourced at https://github.com/SAP/mirrorbench and includes a command-line interface for running and managing user-proxy benchmarking experiments.