InteractiveWorld: A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants

AI summary

Overview

Research area: Human–computer interaction and embodied AI evaluation, specifically egocentric (first-person) multimodal LLM assistants on wearable devices.

Technical level: Advanced. The paper assumes familiarity with multimodal large language models (MLLMs), physics simulators, and agent benchmarks, though the core argument is accessible.

Scope: The paper introduces InteractiveWorld, a simulated, human-centric, closed-loop environment built on Isaac Sim, and InteractiveBench, a 120-task benchmark that scores an AI assistant by whether its natural-language guidance actually leads a simulated human to complete real-world tasks.

What This Paper Is About

Today's benchmarks for egocentric AI assistants are static: a model watches a prerecorded first-person video and answers questions, and nothing it says changes what happens next. But real assistance is a loop — the assistant speaks, the human acts, the world changes, and the assistant must respond to what it now sees. The paper asks whether current MLLMs can actually help a person do physical work, and builds a verifiable simulated environment to answer that question without recruiting real participants.

Key Contributions

1. InteractiveWorld, described as the first verifiable environment for closed-loop interaction between a human and an AI assistant. Built on Isaac Sim with over 8,000 objects and 50 scenes imported from BEHAVIOR-1K, it combines a physical layer (SMPL+H full-body human with a head-mounted camera), a human action layer (collision-aware A* navigation and adaptive manipulation), and a semantic layer that grounds free-form language into 36 action primitives.

2. A role reversal from robot simulators. The assistant never controls the body directly; it can only influence outcomes through natural-language guidance that an autonomous, MLLM-driven Human agent interprets and executes. The paper argues this is the first environment satisfying six properties simultaneously: egocentric view, human actor, guided human, natural-language dialogue, closed loop, and verifiable success.

3. InteractiveBench, described as the first dynamic, closed-loop benchmark for egocentric assistants: 120 human-reviewed multi-turn tasks across six categories (Guidance, Planning, Accessibility, Decision, Prevention, Finding), with success checked automatically against ground-truth object states in the simulator.

4. A 12-model evaluation showing large gaps, plus analyses of efficiency, nine fine-grained capabilities, error types, task horizon, memory augmentation, and varying the Human agent.

Main Findings

• Closed-loop assistance is far from solved. Claude Fable 5.1 has the highest average pass rate at 41.7%, and every other model is at or below 33.3%. The authors verified that an Oracle Assistant reaches a 100% pass rate with the same Human agent, and 0% passes without assistance.

• Long-horizon tasks are the weak point. Best-model pass rates are 20.0% on Guidance and 25.0% on Planning, versus 85.0% on Decision. The authors attribute this to Decision tasks often needing a single scene judgment, while Guidance and Planning require continuous tracking, anomaly detection, and dynamic replanning. Performance also drops as tasks get longer.

• Perception and memory drive most failures. Visual perception and memory errors account for 70.4%–85.7% of first errors across models. For Gemini 3.1 Pro, visual perception accounts for 69.8% of first errors; Kimi K3 fails more on memory than perception (39.2% vs. 33.8%).

• Communication can cap overall success. GLM 5.3 Flash and Claude Fable 5.1 have nearly identical average capability scores (58.2% vs. 59.2%), but GLM 5.3 Flash scores lower on communication (69.8% vs. 79.6%) and completes 11.7 points fewer tasks (30.0% vs. 41.7%). Even for Claude Fable 5.1, 13.6% of first errors come from failing to turn the goal into followable guidance.

• Action efficiency is poor and task-dependent. Nearly all models require far more actions than the oracle, peaking in Accessibility and Guidance at 2.35× to 3.66× the oracle's cost. Claude Fable 5.1 is the least efficient on Accessibility (6.75×) yet efficient on Prevention (1.20×); Kimi K3 matches the oracle on Finding (1.00×) but is worst on Guidance (3.66×).

• Some models show unprompted proactive behavior. Models ask the Human to walk around or look around for a clearer view, flip packages to reveal hidden labels, and query spatial information — behaviors not required by the Assistant prompt.

• Simple memory augmentation helps only partially. Appending a hidden Memory field to the Assistant's responses improves Prevention and Finding by 5–15% across all three tested models, but Planning drops by 5% for two models, and other categories vary by model.

• The Human agent matters. Across all 16 Assistant–Human pairings among four models, rankings stay relatively stable (Fable 5.1 first, GPT-5.6 last), but all four Assistants achieve their highest observed MCR when Gemini 3.1 Pro plays the Human, suggesting its challenge differs from the Assistant role.

Methodology in Plain English

The team built a simulated apartment-style world in Isaac Sim and put a full-body human model in it, with a camera mounted on the head so the AI sees exactly what the person sees. The human body is driven kinematically so it never randomly falls over, and the system pre-checks paths and arm reach before acting, rejecting unsafe actions with a clear failure reason.

Above the physics sit two more layers. The action layer takes high-level commands like "grab the cup" and figures out where to stand, how to walk there (using A* search around obstacles), and how to pose the arm and hand — standing, leaning, or squatting as needed, and re-planning if no route to the object exists. The semantic layer defines 36 commands (from [Walk] and [LookAt] to [Grab], [PutInside], [Pour], [Cook], [Cut]) so an AI assistant can communicate in ordinary language while a separate language model playing the "Human" translates that language into commands.

Each benchmark task starts from a defined scene and warmup actions that give the assistant initial context. Then the two models converse: the assistant chooses per turn to interrupt with guidance, stay silent, or output done; the human replies or issues an action; the simulator executes and renders new frames. Evaluation ends when the assistant says done or a maximum interaction limit is hit, and success is checked against predefined object states (for example, "the pizza is in the oven" or "the fridge is closed"). For the main experiments, Gemini 3.1 Pro served as the fixed Human agent so assistants were compared fairly, with visual input capped at 32 frames per turn (4 frames per action, keeping the 8 most recent plus 24 uniformly sampled from older history).

Why This Matters

Impact on research: The paper argues that static video benchmarks cannot measure what matters for assistance — closed-loop adaptation and compositional skills — and that offline metrics do not reflect online performance. InteractiveWorld provides verifiable, state-based rewards, which the authors note could support training interactive assistants, not just evaluating them.

Real-world applications (drawn from the benchmark's own task categories):

• Guidance: a new lab worker putting away 14 chemical bottles and clearing broken glass, guided using labels, chemical knowledge, and safety rules.
• Accessibility: a blind or low-vision user cleaning a kitchen containing broken glass, food, and dishes, with the assistant describing the scene and reading the cleanup guide.
• Planning: a chef cooking for guests with different food allergies, using one oven and pan within 30 minutes.
• Prevention and Finding: reminding a user before leaving home that the fridge and living-room window were left open, or helping locate black and blue squeegees stored in a lidded box in another room.

Industry relevance: The work comes from Meta with Duke University collaborators, targets wearable egocentric assistants, and is released with code and a project page. It gives device and model builders a reproducible, human-free way to test whether an assistant's advice actually helps — and its efficiency metric (unnecessary instructions equal wasted human effort) is a practical design target.

Future Directions

• Train native memory mechanisms for long-horizon interaction. The authors conclude that a simple inference-time Memory field is insufficient, and that mixed results motivate trained memory rather than prompting tricks.
• Calibrate simulated Human agents against real users. The paper reports that assistant performance shifts with the Human partner and calls for behavioral calibration to better align simulated humans with real people; it also includes (in the appendix) additional Human-agent validation with real human participants.
• Use the environment's state-based rewards for training, not only evaluation, as the authors suggest in the related-work discussion.
• Attack the perception and memory bottleneck directly, since these account for the large majority of first errors, and improve communication so that strong capability scores translate into completed tasks.

Target Audience

Researchers and engineers working on multimodal LLM agents, egocentric and wearable assistants, and embodied AI evaluation; teams that need a reproducible alternative to costly human user studies; and reinforcement-learning researchers interested in environments with verifiable, state-based rewards for language-guided assistance. Readers wanting benchmark design lessons from a large-scale 12-model evaluation will also benefit, though the simulator engineering details assume an advanced background.

Authors’ abstract

People have long dreamed of an assistant that is always there to guide them and remind what they forget, from new dishes to lost keys. Powered by multimodal large language models (MLLMs), egocentric AI assistants on wearable devices are bringing this dream within reach: they see what the user sees and give instructions for user's physical work when necessary. However, testing such assistance is challenging and usually requires real people. The expensive and rare real interaction data leaves a key question unanswered: Can today's MLLMs truly help people in physical world? To answer this question, we propose InteractiveWorld, the first verifiable environment for closed-loop interaction between human and AI assistant. Unlike existing robot-centric simulators, InteractiveWorld lets the assistant guide a human agent who observes the scene, understands instructions, and acts independently. It thus realistically simulates how AI assistant collaborates with human: the assistant instructs, the human acts, and the environment executes actions and updates what they see next. Building on it, we introduce InteractiveBench, the first dynamic benchmark for egocentric AI assistants, with 120 multi-turn tasks across six categories. In each task, the assistant guides the human in real time, and every instruction shapes the outcome, which is scored by verifiable, state-based rewards. Experiments with 12 frontier MLLMs show they are still far from reliable assistants. Even the strongest model, Claude Fable 5.1, achieves at most 20% and 25% on long-horizon Guidance and Planning tasks, respectively. Overall, our work offers a verifiable platform and a rigorous evaluation set, laying a solid foundation for research on interactive MLLM assistants that truly help people in the physical world.