The Pulse
Meta Releases InteractiveWorld, a 120-Task Wearable AI Benchmark
Meta’s InteractiveWorld project pairs a first-person 3D simulator with InteractiveBench, a 120-task test for conversational AI assistants. Its scores check simulated world states, while the software’s hardware and licensing requirements lim
AI.info Team · Written with AI · fact-checked ·
InteractiveWorld’s benchmark contains 120 multi-turn tasks, each scored against what happened in a simulated 3D world rather than an AI assistant’s written answer. Meta’s Facebook Research group has released the system as an open-source project aimed at evaluating assistants that see through a wearer’s eyes and guide a person through real-world tasks.
The release combines InteractiveWorld, a simulator, with InteractiveBench, its task suite. The assistant receives first-person images and conversation; a separate simulated human responds to its guidance by acting in the environment. “Goals are checked against the world state, not judged from text,” the project’s README says. InteractiveWorld project
Six ways to test an assistant
InteractiveBench divides its 120 tasks evenly among six categories: Guidance, Planning, Accessibility, Decision, Prevention and Finding. The scenarios ask an assistant to guide someone through a recipe, plan around time and resource limits, describe a scene for a blind user, make a choice using visual evidence and world knowledge, warn before a mistake, or remember where an object was left.
Each task starts with a simulated scene and may include a short history of what the person did before asking for help. The assistant sees first-person frames and dialogue, then responds with speech, silence or a declaration that the task is complete. The simulated human follows with speech and actions; the simulator renders the changed scene, and the episode ends when the assistant or human finishes or the turn budget runs out.
That setup puts interaction and restraint into the test, not just image recognition. Prevention tasks, for example, ask whether the assistant warns at the right moment without interrupting unnecessarily. Accessibility tasks withhold images from the simulated human, requiring the assistant to communicate what it sees.
Success means changing the simulated world
The repository provides two kinds of scores. A native score checks whether task goals hold in the final world state and reports the fraction of goals met. The release’s official benchmark score uses task-specific criteria that must be adjudicated by reviewing evidence such as transcripts and videos; the documentation lists 464 criteria across six protocols. Runs can store the dialogue, actions, scores and first-person video.
The documentation says the project uses Gemini 3.1 Pro as the fixed simulated human for every assistant in the paper’s setup, a choice intended to make comparisons more consistent. Developers can also connect assistants and human models through hosted services or compatible local endpoints, including servers such as vLLM and Ollama. Keeping the human model fixed matters: a benchmark result measures an assistant paired with that model and the simulator’s rules, not a device operating with an unpredictable person.
Meta reports no model leaderboard or comparative results in the repository. The release offers a way to run evaluations, but its published materials do not establish that a particular assistant performs well on the tasks.
Researchers face a substantial setup
Running the simulator requires Linux, an NVIDIA GPU and about 100 GB of disk space. Meta says it tested on an A100 with 80 GB of memory; one simulator worker uses about 4 GB of GPU memory. The benchmark guide estimates five to 20 minutes per task, depending on task length and model latency, putting a full 120-task run at a meaningful compute and time cost.
Some assets also require separate downloads and registration. The project uses Isaac Sim, OmniGibson, BEHAVIOR-1K scenes, a human body model and walking-motion data; the README says the body-model and motion files cannot be redistributed. The repository carries a CC BY-NC 4.0 license, while those external components remain under their owners’ separate terms.
Those conditions make InteractiveWorld a research testbed rather than a ready-made evaluation for every product team. Its scenes cover 44 BEHAVIOR-1K homes and venues, and its simulated human follows model-generated guidance through scripted actions. Whether results on those indoor scenes predict how a wearable assistant performs with real people, imperfect camera views and changing surroundings remains an open question.
What a passing score can—and cannot—show
InteractiveWorld makes the outcome of a simulated task inspectable: did the person put the right object in the right place, complete the requested action, or avoid a defined mistake? That is more concrete than judging an assistant solely by the fluency of its answer, but it is still a measure of success inside a simulator with a fixed human model.
The next test for the benchmark is whether independent teams can reproduce its scores and whether those scores track performance outside its 44 simulated settings. Until then, a successful run shows that an assistant can complete the benchmark’s defined scenarios—not that it can reliably guide someone wearing glasses through the same task at home.