UserArena: Benchmarking Interactive User Simulation for Long-Horizon Shopping Agents

AI summary

Overview

Research area: Information Retrieval / Recommender Systems — specifically agentic recommendation, user simulation, and offline evaluation of recommendation policies.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning, knowledge distillation, token-level language-model training, and e-commerce A/B testing.

Scope: The paper introduces a production-grounded interactive e-commerce benchmark (UserArena) with a five-level simulator evaluation protocol, plus a training method (On-Policy Trajectory Distillation, OPTD) for building more faithful long-horizon user simulators.

What This Paper Is About

Training and evaluating "agentic" shopping assistants requires a recommendation world model — a simulated shopping environment plus a simulated user — so that policies can be tested offline instead of through expensive online experiments. Existing resources either lack long-horizon, step-aligned real-user trajectories that align profiles, screenshots, product context, and actions, or they evaluate simulators only by next-action agreement rather than by whether simulated trajectories preserve a user's behavior over many sessions and reproduce real A/B outcomes. The paper addresses both gaps by releasing a large real-user benchmark with an interactive environment and a five-level evaluation protocol, and by proposing a distillation method that makes simulators behave more like real shoppers over long rollouts.

Key Contributions

1. UserArena benchmark. A real-world e-commerce interactive benchmark built from a large Asian e-commerce platform, containing 64,729 session trajectories from 11,010 users, covering 566,102 items and 13 action types, with user profiles, visual observations, product context, and actions aligned at every step. It links sessions from the same user into cross-session trajectories (59,378 chronologically ordered sessions from 5,768 users, averaging 10.29 sessions and 86.36 actions per user) and supports offline validation against a controlled production A/B test.

2. A five-level evaluation protocol. Simulator fidelity is scored progressively: L0 environment compliance (valid-action rates under Replay and Rollout), L1 local behavioral fidelity (F1 and accuracy on next actions), L2 sequential behavioral fidelity (decision-mix Jensen–Shannon divergence and trajectory error), L3 personalized population fidelity (population fidelity across user groups), and L4 outcome fidelity (click-through-rate error, add-to-cart-rate error, and terminal-state JSD).

3. On-Policy Trajectory Distillation (OPTD). A training method that extracts a trajectory-level "intent" from each complete training trajectory and gives it to a frozen teacher as privileged context, providing dense token-level targets on the student's own rollouts rather than on logged prefixes. A per-token trust region (via KL-divergence budget against an intent-free reference distribution) bounds how much the intent may shift each target, and a grounding loss keeps step-level supervision from recorded responses.

4. Benchmarking of representative simulators. A controlled comparison of six text-based simulators (RecAgent, Agent4Rec, Shop-R1, Customer-R1, AlignUSER, SimUSER) and two GUI-based simulators (A/B Agent, STA), all using Qwen3.5-2B, plus ablation and source-blinded Turing-style human evaluation of OPTD.

Main Findings

• Single-session dominance of OPTD: OPTD ranks first on all eight reported L1–L4 metrics in the single-session setting, reaching F1 0.4178, accuracy 0.3204, DM-JSD 0.0156, trajectory error 1.72, population fidelity 0.8692, ΔCTR 0.0026, ΔACR 0.0430, and T-JSD 0.0387.

• Cross-session results are mixed but favorable: In the cross-session setting OPTD ranks first on five of the eight L1–L4 metrics and second on two more, with the best L2 sequential fidelity (DM-JSD 0.0284, trajectory error 0.72) and the highest L3 population fidelity (0.8929). STA leads on macro-F1 (0.3556) and ΔCTR error (0.0099), showing that its reinforcement-learning post-training helps selected metrics but not consistently across levels.

• OPTD best reproduces a real production A/B effect: In an offline comparison of the same ranking-policy pair, with 10,000 frozen users split evenly across arms and the effect measured as between-arm ΔPV-CTR, OPTD attains an effect ratio of 0.713× (effect error 0.287). Other simulators fall far short: Shop-R1 0.162×, Customer-R1 0.191×, AlignUSER 0.020×, STA 0.297×, Qwen3.5-2B 0.004×.

• General-purpose model scale does not buy behavioral calibration: GPT-6-Sol and GPT-6-Luna overestimate the online response, with effect ratios of 1.576× and 2.154× respectively (effect errors 0.576 and 1.154).

• GUI simulators can be brittle on a fixed action protocol: A/B Agent shows notably low valid-action rates (Replay 0.4113, Rollout 0.6302 in single-session; 0.5182 and 0.5777 cross-session), which limits its downstream fidelity despite being competitive on ΔACR error.

• Every OPTD component contributes: Ablating on-policy sampling, intent, the trust region, the grounding term, or the SFT warm-up all reduce performance on at least one selected metric, with removing SFT ("w/o SFT") degrading most markedly.

• Human observers are fooled more often by OPTD: In a source-blinded Turing-style evaluation over 200 held-out trajectory pairs with three annotators, OPTD achieves the lowest Human win rate (correctly spotted as machine-generated) and the highest Model win rate (real trajectories mistaken for simulated) among OPTD, Customer-R1, STA, and A/B Agent.

• Behavioral composition is highly skewed: Feed-page downward scrolling is the most frequent action at 38.34%, followed by item clicks at 12.76%, session termination at 11.21%, returns to the feed at 10.62%, and item-page scrolling at 10.08%; add-to-cart is 3.90% and purchase 1.02%. Apparel accounts for 54.90% of the 73,717 item clicks across 41 categories.

• Interactions are long-tailed and long-horizon: 92.0% of the 67,267 clicked items appear only once, and the most-clicked 1% of items account for only 3.23% of clicks. Across 53,610 adjacent session pairs, 57.8% revisit a product category from the preceding session, with gaps ranging from minutes to more than 12 hours.

Methodology in Plain English

Building the benchmark. The authors took de-identified interaction logs from a large e-commerce platform and re-created the shopping interface offline. They rebuilt the two-column recommendation feed and item-detail pages in a React environment and batch-rendered the screens, so no real user screenshots are released. A simulator sees the current page and history, then emits a short rationale plus one structured action. Because the environment is pre-rendered, a simulator can click any exposed item — including ones the real user never clicked — and the corresponding detail page opens, letting the environment support counterfactual exploration rather than replay only. Actions cover 13 types, from feed navigation and product inspection to add-to-cart and purchase. Sessions belonging to the same user are chained chronologically to form long-horizon trajectories.

Evaluating simulators. Two modes are used. Replay hands the simulator the recorded state at each step and asks for the next action, isolating local prediction. Rollout lets the simulator act freely and drive its own state until the session ends, testing whether free-running sessions look like real ones. L0 is scored in both modes, L1 in Replay, and L2–L4 in Rollout.

Training a better simulator (OPTD). Three problems motivate the method. First, supervised fine-tuning (SFT) trains on ground-truth prefixes but at inference the model conditions on its own possibly-wrong prefixes (exposure bias). Second, exact-match targets penalize plausible alternative actions and ignore trajectory-level quality (sparse supervision). Third, RL methods like GRPO assign a single scalar reward per response, which is coarse credit assignment. OPTD addresses all three:

• A large language model (Qwen3.5-397B) acts as an extractor that summarizes each training trajectory into a structured "intent" record — preferences, concerns, information needs, comparison criteria, conditional tendencies — deliberately excluding concrete actions, product IDs, prices, outcomes, and step indices, so it cannot serve as an answer key.
• The student policy samples a response at a real state. A frozen teacher then scores every token of that response, both with the intent and without it.
• The two teacher distributions are interpolated in log-probability space, and at each token the largest interpolation weight is chosen whose KL divergence from the intent-free reference stays within a fixed budget. Guidance that would push too hard is pulled back. This is the trust region.
• The student matches this regularized target with a generalized Jensen–Shannon divergence averaged over tokens, and this distillation loss is combined with a standard teacher-forced grounding loss on the recorded response.

Validation. Beyond automatic metrics, the authors run a source-blinded human "Turing test": three annotators see an anonymized, order-randomized real–simulated pair and must say which is machine-generated or answer "Unsure." They also compare simulated policy effects against a real controlled A/B test using the same ranking-policy pair.

Why This Matters

Impact on research. Offline evaluation of recommendation policies is only as trustworthy as the user simulator behind it. This paper makes two moves that change how the field can judge simulators: it supplies step-aligned real trajectories with visual context and full action spaces over multiple sessions, and it adds an outcome criterion — whether a simulated policy comparison agrees with a production A/B result — that no prior benchmark in the comparison table provides. The negative result that a large general-purpose model (GPT-6-Luna, 2.154× effect ratio) is worse calibrated than a small distilled one is an important corrective to the assumption that scale alone solves simulation fidelity.

Real-world applications:

• Cheap pre-screening of ranking-policy changes. Teams can rank candidate policies in simulation before committing traffic to an online A/B test.
• Reinforcement-learning training of shopping agents. A faithful simulator supplies the reward signal and interaction environment needed for RL without exposing real users to untested policies.
• Stress-testing long-horizon personalization. Because sessions are chained per user, developers can check whether a policy erodes or preserves a shopper's preferences across repeated visits.
• Regression testing of conversational and GUI shopping assistants. The GUI-based simulator comparison shows which agents can actually operate a fixed interface action protocol.

Industry relevance. The work originates from a collaboration including a large e-commerce platform, is supported by the Alibaba Innovative Research Program, and is grounded in a real controlled A/B test rather than synthetic dynamics — which is precisely the evidence practitioners need before trusting offline results.

Future Directions

• A living leaderboard. The authors state they will build a leaderboard platform around UserArena for standardized, continuously updated simulator evaluation.
• Stronger long-horizon simulators. They explicitly call for simulators that preserve evolving preferences and behavioral consistency across sessions — the cross-session results show OPTD is strong but not uniformly best.
• Broadening beyond one platform and one domain (a logical extension, not stated as future work): all evidence here comes from a single e-commerce platform with apparel dominating 54.90% of clicks, so transfer to other catalogs, languages, and interaction surfaces is untested.
• Validating more than one policy pair (a logical extension): the production A/B validation covers a single ranking-policy pair, so how reliably effect ratios like 0.713× generalize across policy families remains an open question.
• Closing the remaining fidelity gap. OPTD's effect error is 0.287, meaning the simulated effect is still roughly 30% off the online value — a substantial residual that future simulators would need to reduce.

Target Audience

Researchers and practitioners in recommender systems, information retrieval, and LLM-agent evaluation who need offline, counterfactual, or reinforcement-learning-based evaluation of recommendation policies. It is most useful to those building or auditing user simulators (the five-level protocol and OPTD are directly actionable), to industry teams weighing whether simulation can substitute for parts of an A/B testing pipeline, and to readers interested in distillation methods that handle exposure bias and sparse supervision over long generation horizons. Readers without background in RL or language-model training will find the benchmark and evaluation protocol accessible but the OPTD section demanding.

Authors’ abstract

Agentic recommendation pursues shopper goals through sequences of context-dependent decisions. Developing and evaluating such policies requires faithful recommendation world models that capture both the interactive shopping environment and users' behavioral responses over long interactions. Existing resources rarely provide long-horizon, fine-grained real-user trajectories or support trajectory-level evaluation and offline validation against production A/B results. We introduce UserArena, a real-world e-commerce interactive benchmark containing 64,729 session trajectories from 11,010 users, spanning 566,102 items and 13 action types. UserArena pairs these trajectories with an interactive environment and supports offline validation against outcomes from a controlled production A/B test, enabling five-level evaluation from output validity and local decisions to sequential behavior, personalization, and outcomes. To improve simulator fidelity over these long-horizon interactions, we further propose On-Policy Trajectory Distillation (OPTD), which uses trajectory-level intent to provide dense teacher targets on student-generated rollouts. Experiments show that OPTD achieves the strongest overall fidelity among the evaluated simulators under UserArena's evaluation protocol and most closely reproduces the production policy effect in offline simulation.