Interactive Visual Gym: Evaluating Visual Interaction Models with Simulated Users in Interactive Worlds

AI summary

Overview

Research area: Computer Vision / multimodal interactive agents — specifically, benchmarks and training environments for visual interaction models (assistants that watch a live video stream of a person doing a physical task and talk them through it).

Technical level: Intermediate. The paper is readable without specialist background, but it assumes familiarity with vision-language models (VLMs), benchmark design, simulated-user evaluation, and reinforcement learning from preference-style rewards.

Scope: The paper introduces a closed-loop simulation framework — three executable task worlds plus persona-conditioned simulated users — instantiated as a 315-scenario benchmark (IVG-Bench), validates it with a 45-participant human study, and demonstrates that the same worlds can be used as a reinforcement-learning environment.

What This Paper Is About

Assistants that guide people through real-world physical tasks (cooking, tidying, changing phone settings) are usually evaluated by replaying a recorded video and scoring the model's replies against reference answers. That setup is fundamentally broken: in reality, what the assistant says changes what the user does next, so a reply that looks wrong in isolation might actually rescue the task, and a reply that looks right might be useless or too late. The paper builds a gym where a simulated human actually performs the task in a simulated world, reacts to the assistant's guidance, makes realistic mistakes, and can ignore or question advice — so the assistant's effect on task outcome and user experience can be measured in a closed loop.

Key Contributions

1. Interactive Visual Gym, a closed-loop framework for evaluating and training visual interaction models by jointly simulating executable task environments and human users, with world state tracked as scene graphs so progress is measured independently of how a task was executed.
2. Three domain simulators with human-communicable action spaces: CookSim (cooking, built from scratch in Three.js/WebGL, top-down and egocentric 3D), VHSim (an adaptation of VirtualHome with a Unity backend for apartment tasks), and ScreenSim (an iPhone 17 Settings interface in HTML/CSS rendered in Chromium and captured via Playwright). Commands are phrased at the level of everyday speech, e.g. "go to the nearest pan," "place the book on the table," "select Background App Refresh."
3. A persona-conditioned user-simulation harness that role-plays a human with structured world-state access, revises plans in response to guidance, and injects scheduled out-of-plan events (execution errors such as repeating, forgetting, skipping a step, wrong plating order, wrong ingredient, adding unwanted ingredients, plating raw food, unattended cooking; questions about action order, counts, elapsed time, duration, event recall; proactive request-to-notify; and plan changes). Personas vary along four axes with three levels each: skill level, trust in AI, execution speed, and interruption preference.
4. IVG-Bench, a 315-scenario benchmark (150 cooking, 75 household, 90 mobile-interface), plus a human validation study and a proof-of-concept GRPO training experiment showing the environments supply a usable learning signal.

Main Findings

• Even the best model is far from reliable. Gemini 4 Argon is the top performer but completes only 53.4% of tasks within the time limit, averaged across the three environments. It also trails GPT-6 Astra, GPT-6.1 Sol, and Gemini 3.8 Flash on interaction quality, with notable weaknesses in factual grounding and actionable guidance.
• Household tasks are near-solved by nobody. In VHSim, the best in-time success rate in the main table is 15.6% (Gemini 4 Argon and GPT-6.1 Sol), against 60.9% in CookSim and 84.8% in ScreenSim for the same model.
• Open-weight assistants lag on both axes. In-time success in CookSim ranges from 5.6% (Molmo2-8B) to 40.7% (Muse Glimmer), against 60.9% for Gemini 4 Argon; open-weight streaming models are weaker still (Proact-VL 2.4%, StreamingVLM 5.3%).
• High task success does not imply good guidance. Muse Glimmer completes more CookSim tasks than GPT-6 Astra (40.7% vs 40.2%) while scoring far lower on interaction quality (21.6 vs 76.0 overall), showing that outcome-only metrics can hide poor assistance.
• Strong streaming models trade reasoning for latency. Gemini 3.8 Live reaches high average interaction quality yet completes fewer tasks than GPT-6 Astra and Gemini 3.8 Flash, consistent with a low-latency-versus-planning trade-off.
• Temporal question answering during a live task is surprisingly hard. Overall question-answering/proactive scores in CookSim: Gemini 4 Argon 79.0, GPT-6 Astra 78.6, GPT-6.1 Sol 74.3, Gemini 3.8 Flash 59.4, GPT-Realtime 2.1 46.5, Qwen3.8-27B 38.7, and single digits for several open streaming models (StreamingVLM 3.4, Proact-VL 1.3). Action counting is the weakest sub-skill even for frontier models (Sol 22.9, Astra 34.8, Argon 52.8), and open-weight models often lose track of the question entirely across multi-turn dialogue.
• Latency reshuffles the leaderboard. With measured assistant latency, Gemini 4 Argon's CookSim in-time success drops from 60.9% to 33.1%, while Muse Glimmer barely changes (42.7%, +2.0) and ends up leading in CookSim. All four frontier API models lose success in both CookSim and VHSim, with declining error prevention explaining the loss — warnings arrive after the user has committed the error.
• The interaction-quality metric tracks human preference. Across 45 participants, 3 anonymized assistants, and 3 rounds in CookSim, the metric agrees with participant rankings in 78.5% of participant–assistant-pair comparisons, with a pooled Kendall's τ of +0.57.
• Simulated users reproduce most, but not all, human behavior. Similarity is high for guidance requests (0.98), utterance length (0.89), disagreement (0.88), ignored interruptions (0.82), action efficiency (0.82), suggestion uptake (0.77), and task success (0.56), but near zero for speech rate (0.05), responsiveness (0.05), and action parallelism (0.00). Simulated users speak to the assistant far more often than real participants, plausibly because the user agent is invoked very frequently to reconsider whether to talk.
• The worlds work as an RL environment. With Qwen3.5-4B trained by GRPO on CookSim, after 12 RL updates in-time success rises by 8.9 percentage points on held-out validation scenarios with a Gemini 3.7 Flash user (33.3 → 42.2) and by 3.3 points with a Gemini 3.8 Flash user (64.4 → 67.8). Interaction quality also improves (29.6 → 31.1 and 24.3 → 30.2), though detection recall drops (49.0 → 34.9 and 43.0 → 28.7).

Methodology in Plain English

The authors built a video-game-like world for each domain where a virtual person can actually do the task: walk around a kitchen, pick up a pan, cook a patty, plate a cheeseburger. The world keeps an internal map of what is where and in what state (a scene graph), so the system can tell whether the task is genuinely done, regardless of the order in which the user did things. Crucially, the actions available to the virtual user are phrased the way a person would say them out loud, not as joystick movements — a language model translates "the pan on your left" into the right object for the simulator.

A second VLM is cast as the user. It plans and acts on its own, receives the assistant's messages, and decides whether to follow, ignore, or question them, all filtered through an assigned persona (how skilled it is, how much it trusts AI, how fast it moves, how much interruption it tolerates). The harness can also take over the user's hands at predefined moments to stage mistakes or plan changes, which forces the assistant to notice and respond.

Time is handled honestly: each simulation tick is given a real-world duration (1.2 seconds for a one-meter VirtualHome step), and the assistant's measured inference latency is converted into a number of ticks of delay, so a slow reply arrives after the world has moved on. Episodes are scored on whether the task finishes within a budget of twice the error-free reference time plus an event allowance, on whether triggered errors were caught and whether they were prevented before the point of no return, and on five binary rubric dimensions of interaction quality — factual grounding, situational relevance, actionable guidance, user intent uptake, and guidance conciseness — judged by two LLM judges against the true world state rather than against a reference script.

To check the setup is meaningful, they ran a human study in CookSim, and to check it can teach, they ran a small GRPO experiment with a shaped reward: 1 for success, +0.30 for preventing an error, −0.02 per false flag (capped at 15), and −0.05 for excess turns (capped at 1), training on 120 of CookSim's 150 scenarios with 24 episodes per update.

Why This Matters

Impact on research: The paper argues that offline replay — matching replies to a fixed recorded trajectory — cannot measure whether assistance actually helps, and provides an alternative with response-contingent rollouts, engine-verified outcomes, and state-grounded dialogue judging. It also shows the same infrastructure can serve as a training environment, not just a test set.

Real-world applications:

• Cooking and recipe assistants that watch a live camera feed, catch a missed or misordered step, and recover the dish before it is ruined.
• Accessibility and eldercare support, where an assistant guides someone through household chores or device setup and must adapt to the user's skill and preferred level of interruption.
• Mobile and consumer-device onboarding, using the ScreenSim-style settings walkthroughs to check that guidance is factually correct and actionable on a real interface.
• Remote expert support and training, where the same setup could evaluate how well an expert-in-a-box handles novice mistakes and questions in real time.

Industry relevance: The finding that response latency collapses task success for frontier API models (Gemini 4 Argon: 60.9% → 33.1% in CookSim) directly implicates serving infrastructure, not just model quality, as a design variable — and the fact that a small open-weight model improves with only 12 RL updates on 120 scenarios suggests a practical path to specializing compact models for interactive assistance.

Future Directions

• Scale the RL proof-of-concept. Training covered only CookSim, 120 scenarios, and 12 updates, with a single 4B model; whether the same signal scales to VHSim and ScreenSim and to larger models is untested.
• Close the human-simulation gap in communication. Simulated users talk far more often than real participants, and speech rate, responsiveness, and action parallelism similarities are 0.05, 0.05, and 0.00 — improving fidelity here would make the loop more trustworthy.
• Train and evaluate for latency, not around it. Since measured latency reverses the CookSim ranking and degrades error prevention, models may need to be optimized for timely, short-horizon intervention rather than for offline response quality.
• Fix the specific weak spots the benchmark exposes — temporal action counting and event recall in multi-turn dialogue, and the drop in error-detection recall observed after RL — and broaden validation beyond a single cooking simulator and a 45-participant study.

Target Audience

Researchers and engineers building multimodal assistants for live, physical, or screen-based tasks; benchmark designers interested in closed-loop, simulated-user evaluation; HCI researchers studying persona-adaptive and interruption-sensitive assistance; and applied teams at companies shipping real-time guidance features who need to reason about the joint effects of model quality, latency, and interaction style.

Authors’ abstract

Visual interaction models that support users during ongoing physical tasks require evaluation in closed loop, where assistant responses can change subsequent user behavior and task outcomes. Existing benchmarks largely rely on offline replay and therefore cannot capture these effects. We introduce Interactive Visual Gym, a framework for closed-loop evaluation and training of visual interaction models through joint simulation of executable task environments and human users.…