SWITCH: Benchmarking Modeling and Handling of Tangible Interfaces in Long-horizon Embodied Scenarios

AI summary

Overview

• Research area: Multimodal computer vision and embodied AI — specifically benchmarking how vision-language models understand and operate "tangible control interfaces" (switches, appliance panels, remotes, elevators, embedded GUIs) in real-world first-person video.
• Technical level: Advanced. The paper assumes familiarity with multimodal benchmarks, video QA, LLM-as-judge scoring, and video generation models.
• One-sentence scope: SWITCH is a 1,170-video, real-world egocentric benchmark that tests whether models can close the loop on interface interaction — understand the interface, predict actions and their consequences, and verify or recover from the outcome.

What This Paper Is About

Everyday environments are full of physical controls, and using them requires more than recognizing a button: an agent must read the instruction, pick the right control, anticipate what the action will change, watch the resulting state change over time, and confirm the goal was actually met. Existing benchmarks mostly test open-loop perception or single-step action selection, so they miss the continuous cycle of acting, observing feedback, and correcting. SWITCH builds that full closed-loop cycle into one benchmark, using real egocentric videos with dense temporal annotations of instructions, actions, state transitions, outcomes, and recovery behaviors.

Key Contributions

1. A real-world benchmark for TCI-centered closed-loop interaction. SWITCH is built from 1,170 temporally interactive egocentric videos spanning nine domains of everyday tangible control interfaces, with language-grounded annotations of fine-grained actions, state transitions, task outcomes, and recovery behaviors.
2. A unified evaluation protocol across three formats. The benchmark supports Multiple-Choice QA (MCQ), free-form OpenQA graded by a strict rubric-based binary LLM judge, and Interactive Video Generation for probing generative world modeling.
3. A structured three-stage, eight-task capability framework. Evaluation is organized into Interface Understanding (Task Understanding, State Perception, UI Recognition), Action & Consequence Reasoning (Action Reasoning, Sequential Action Quantification, State Transition Prediction), and Closed-loop Reflection (Outcome Verification, Recovery Reasoning).
4. A systematic evaluation of frontier multimodal and video generation models. The paper benchmarks eleven proprietary and open-weight multimodal models plus six video generation models, with both automatic judging and human evaluation, revealing persistent weaknesses in fine-grained visual-temporal perception, outcome verification, and failure recovery.

Main Findings

• Closed-loop interaction remains far from solved. On the diagnostic MCQ subset — which removes items already answered correctly by Qwen2.5-VL-7B-Instruct — the strongest model reaches only 43.72% overall accuracy and the best open-weight model reaches 30.66%.
• UI recognition outpaces state perception. GPT-5.5 scores 65.38% on UI Recognition but only 25.82% on State Perception, a 39.56-point gap; across models, the best State Perception score is below 40%. Recognizing where a control is does not imply understanding what state the device is in.
• Temporal action tracking is brittle. Sequential Action Quantification is the hardest action skill: the best MCQ score is 40.20% and the best OpenQA score is 36.7%, suggesting models struggle to maintain temporal consistency over longer video segments.
• Verification and recovery expose the closed-loop gap. The top MCQ scores are 44.76% for Outcome Verification and 43.67% for Recovery Reasoning; under OpenQA both abilities drop below 10%.
• OpenQA reveals reliance on answer-option scaffolding. The best MCQ overall accuracy is 43.72%, while the best OpenQA accuracy on the no-options subset is only 18.9%; GPT-5.5 drops from 39.77% in MCQ to 16.3% in OpenQA. Without candidate options, models produce partial, vague, or contradictory responses (Table 6: for example, MIMO-V2.5 shows 13.1% exact matches and 59.1% complete misses).
• Generative world modeling favors plausibility over correctness. Across the 1,170 generated videos, execution_observability is consistently the strongest dimension (3.76 average for image-to-video, 4.10 for video-to-video), while scene_fidelity (2.04 and 2.70), control_correctness (2.26 and 2.72), and procedure_correctness (2.34 and 2.62) remain substantially weaker. Doubao-Seed 2.0 performs best in both settings.
• Video conditioning helps only modestly. Video-to-video overall averages 2.18 versus 1.92 for single-image conditioning, so observing a short prefix aids continuity but does not solve device-state tracking or causal task structure.
• Human evaluation agrees with the automatic judge. On 160 videos from four categories, each rated by three blind annotators, humans assign higher absolute scores and slightly reshuffle the image-conditioned ranking, but identify the same bottleneck: appearance and visible interaction are preserved better than control-goal binding and correct procedure order.
• Five recurring generative failure modes. Models violate physical constraints, misidentify the relevant control, produce unnatural motor behavior, break procedural logic (repeating actions, ignoring preconditions, unsupported outcomes), and hallucinate UI elements or objects — often while looking convincing at a glance.

Methodology in Plain English

The authors first collected 1,170 egocentric videos of real people interacting with everyday control interfaces across nine domains, instructing collectors to record complete task-oriented segments — preparation, manipulation, waiting, inspection, and outcome checking — rather than isolated button presses. Annotators then densely labeled each video with both interface-level evidence (indicator lights, display text, selected modes) and environment-level evidence (lighting changes, water flow, temperature shifts), explicitly distinguishing task-execution actions such as pressing or rotating from verification actions such as checking system status, and marking failure and recovery behaviors.

From those annotations the team derived evaluation tasks aligned with the three-stage framework and three parallel formats: multiple-choice questions, open-ended VideoQA, and generative world-model tasks. MCQs use plausible, source-aware hard negatives plus generative distractors made with Seedance, and multi-select items are scored with strict exact set matching and no partial credit. OpenQA answers are graded by a strict binary LLM judge using task-specific rubrics that penalize hallucinated details, require minimal sufficient specificity (for example, distinguishing "red" versus "green" buttons), and enforce temporal consistency; refusals and formatting failures score zero.

The pipeline initially produced more than 27,000 candidate QA pairs and generative items. After manual review and automatic validation, items that were ambiguous, duplicated, broken, or answerable without visual evidence were removed, yielding a final benchmark of 8,797 visually grounded instances, split strictly by source video to prevent leakage. A diagnostic subset was additionally derived by isolating items a representative baseline (Qwen2.5-VL-7B-Instruct) handles with near-saturated performance.

For the generative track, the authors sampled 20 image-conditioned and 5 video-conditioned instances per category, giving 180 and 45 inputs, and generated 1,080 image-conditioned plus 90 video-conditioned videos. Prompts constrain task type, next action, stop condition, required visible evidence, viewpoint, device layout, and object identity. Every generated video is scored 1–5 by Gemini 3.1 Pro acting as a VLM judge on scene fidelity, control correctness, procedure correctness, and execution observability, with a subset also assessed by human raters.

Why This Matters

Research impact. SWITCH reframes physical interface use as a multimodal reasoning problem centered on action consequences and feedback. It provides a diagnostic testbed that separates "can the model see the interface" from "can the model tell whether the action worked" — a distinction the reported results show current models fail at, since top MCQ scores sit in the low-to-mid 40s and OpenQA verification and recovery fall below 10%.

Real-world applications:

• Home and building automation, where an assistant must confirm that a light, oven, or thermostat actually reached the requested state rather than assuming success.
• Assistive and elder-care robotics, where an agent operating an appliance panel must detect an incomplete or failed manipulation and re-plan.
• Industrial and facility control, where panels and elevators involve delayed or spatially displaced effects that fall outside the current camera field of view.
• Video and simulation generation for training data, where the benchmark's finding that models produce convincing but causally wrong interactions matters for synthetic data quality.

Industry relevance. The benchmark targets the exact capability gap that blocks deployment of multimodal agents in physical settings: reliable outcome verification and error recovery. Its strict multi-select scoring, hard-negative distractors, and rubric-based OpenQA judge give model developers a discriminating signal that option-selection accuracy alone does not provide. The public release of benchmark resources and a leaderboard lowers the barrier for comparing systems on the same closed-loop criteria.

Future Directions

• Broaden coverage. The authors note the current release is concentrated in a limited set of environments and interface styles and may underrepresent region-specific, language-specific, and industrial control settings; they plan more diverse and international data collection.
• Move from offline to online evaluation. SWITCH currently evaluates recorded videos, QA, and generative continuation, but not real-time embodied challenges such as actuation errors, latency, safety constraints, and error accumulation; future versions aim to include real-machine interaction data.
• Strengthen verification and recovery coverage. Verification- and recovery-related cases are harder to collect because they require delayed effects, failed attempts, or multi-step corrections, and are relatively less abundant than standard task-execution examples; more failure-driven and delayed-outcome data is planned.
• Improve generative evaluation. The automatic protocol relies on a VLM-as-Judge and human evaluation covers only a subset of categories and models, so both may miss subtle causal or long-horizon procedural errors; the authors plan expanded human evaluation, better automatic judging, and more direct evaluation of generated interaction trajectories.

Target Audience

Researchers and engineers working on multimodal large models, embodied agents, video understanding, and world models; benchmark designers interested in closed-loop evaluation protocols and rubric-based judging; and product teams building assistants or robots that must operate physical controls and verify that the intended outcome was achieved. Readers need some background in vision-language evaluation to follow the tables, but the three-stage framework itself is described without heavy jargon.

Authors’ abstract

Tangible control interfaces (TCIs), such as appliance panels, remotes, elevators, and embedded GUIs, are a fundamental component of everyday human-built environments. Interacting with these interfaces requires agents not only to ground language in visual observations,but also to execute actions, track temporally evolving state changes, and verify whether intended outcomes have been achieved. However, existing benchmarks predominantly evaluate open-loop perception or single-step action execution, failing to capture this continuous cycle of interaction, feedback, and correction. We introduce SWITCH, a benchmark for closed-loop interactive reasoning with TCIs in realistic egocentric environments1. SWITCH comprises 1,170 temporally interactive videos across diverse functional categories, providing structured annotations of instructions, actions, state transitions, outcomes, and recovery behaviors over time. To probe generative world modeling, SWITCH also evaluates video generation models on interaction-centered tasks using both LLM-as-judge and human evaluation2.Experiments with frontier proprietary and opensource multimodal models reveal persistent weaknesses in fine-grained visual-temporal perception, outcome verification, and error recovery, highlighting SWITCH as a testbed for closed-loop embodied intelligence.