RoboInter: A Holistic Intermediate Representation Suite Towards Robotic Manipulation

AI summary

Overview

Research area: Robotics — specifically vision-language-action (VLA) models and vision-language models (VLMs) for robot manipulation, with a focus on intermediate representations (subtasks, traces, bounding boxes, affordances, etc.) that sit between high-level plans and low-level actions.

Technical level: Intermediate. The paper is readable by anyone familiar with basic concepts of robot learning and multimodal models, but the benchmark tables and the plan-then-execute terminology assume some background in VLA research.

Scope in one sentence: The paper introduces RoboInter, a unified suite of annotation tooling, a 230k-episode densely annotated manipulation dataset, a 29-category embodied VQA benchmark, and a family of plan-then-execute VLA models, and uses them to measure how intermediate representations affect robot manipulation performance.

What This Paper Is About

Robot manipulation datasets usually pair camera images with an overall language instruction and the robot's actions, but they rarely provide the fine-grained intermediate steps — subtasks, object boxes, grasp poses, motion traces — that modern "plan-then-execute" systems need as supervision. Annotating those intermediates by hand is expensive, and existing automated attempts are limited in scale, scene diversity, or alignment with the actions that were actually executed.

RoboInter's goal is to close that gap by building one integrated resource: a semi-automatic annotation GUI, a large-scale per-frame annotated dataset, a benchmark that tests whether VLMs can understand and generate those intermediates, and VLA models that use them.

Key Contributions

1. RoboInter-Tool — a lightweight GUI for semi-automatic, per-frame annotation of embodied videos. It supports task decomposition into 15 predefined primitive skills, contact-frame recording, object segmentation/tracking routed to SAM2, and end-effector trace reconstruction, with a re-annotation and inspection step to limit the effect of segmentation or tracking errors.

2. RoboInter-Data — a dataset of over 230k manipulation episodes across 571 distinct scenes, drawn mainly from Droid and RH20T (with additional in-the-wild material from OXE), providing dense per-frame annotations over more than 10 categories of intermediate representations. Reported totals include nearly 61M-frame object grounding annotations, about 70M-frame gripper trace annotations, 190k affordance box and placement proposal annotations, and nearly 760k language clip annotations, spanning 6 robot arm types and 15 primitive skills.

3. RoboInter-VQA — a curated embodied VQA benchmark and dataset with 9 spatial and 20 temporal categories, organized by intermediate representation type (spatial vs. temporal) and target capability (understanding vs. generation). Reported scale: approximately 1M spatial generation entries, 172k spatial understanding entries, 131k temporal generation entries, and 935k temporal understanding entries, with 7,246 videos held out as an evaluation pool.

4. RoboInter-VLA — an integrated plan-then-execute framework with a VLM-based Planner and a VLA Executor, supporting three variants (RoboInter-IC-E2E, RoboInter-EC-E2E, and RoboInter-Modular) plus textual and visual-prompted modular forms (RoboInter-Te-Modular, RoboInter-Im-Modular), bridged by a Flexible Chain-of-Thought (F-CoT) that combines multiple intermediate representations.

Main Findings

• Third-party benchmarks improve sharply over base and embodied VLM baselines. RoboInter-Qwen-3B scored 80.0% on RoboRefIt-test and 43.3 on RoboVQA; RoboInter-Qwen-7B scored 85.6% and 74.4; RoboInter-LLaVAOV-7B scored 89.3% and 74.5. The paper reports that at the 3B scale RoboInter-Qwen-3B gained 49.1% over RoboBrain2.0 on RoboRefIt and 12.7% on RoboVQA, and at 7B scale the gains were 76.8% and 42.8%. RoboInter-Qwen-7B ranked second overall on Refcoco among the compared models, a 27.4% relative improvement over Robobrain2.0-7B. General capabilities were described as "slightly affected" but relatively stable.

• General and closed-source VLMs struggle on RoboInter-VQA, while trained RoboInter models do well. For spatial generation, most untrained models scored typically below 40% (ACC@IoU > 0.1). RoboInter-LlavaOV-7B reached 82.9% on Object Grounding and 46.3% on Grasp Affordance; RoboInter-Qwen-3B reached 61.6% on Gripper Grounding. On temporal trace generation (DTW, lower is better), RoboInter-LlavaOV-7B scored 299 against 541 for Robobrain-2.0-7B and 1702 for QwenVL2.5-7B.

• A pretrained Planner improves the Executor's action generation. In the In-the-Wild open-loop evaluation, mOLS rose from 0.3086 for the Vanilla baseline to 0.3218 for RoboInter-IC-E2E and 0.3340 for RoboInter-EC-E2E.

• Explicit intermediate representations beat implicit ones, and decoupling planning from execution helps further. RoboInter-Te-Modular achieved the best non-oracle mOLS of 0.3543, ahead of EC-E2E (0.3340) and IC-E2E (0.3218). The oracle setting — where ground-truth intermediates guide the Executor — reached mOLS 0.3861, which the authors read as evidence that the annotations are informative and stable. Using the original Qwen2.5-VL as a zero-shot Planner (QwenVL+Executor) reached only 0.3102, described as not comparable to the other non-E2E models.

• Finer-grained intermediate representations produce larger gains. The ablation over representation types showed coarse labels (Subtask, Primitive Skill) gave only marginal improvements, while spatially grounded ones (Object Box, Gripper Box, Affordance) gave substantially larger gains. Trace produced the largest single jump, taking mOLS from 0.3435 to 0.3861.

• Open-loop and closed-loop results diverge in an instructive way. In 15 in-distribution and 15 out-of-distribution real-world trials on a Franka Research 3 arm, RoboInter-IC-E2E averaged 77.3% success in-distribution versus 65.0% for Vanilla, and 58.3% versus 38.3% out-of-distribution. RoboInter-EC-E2E had a lower ID rate (68.3%) than IC-E2E (77.3%) but a higher OOD rate (60.0% vs. 58.3%), with an ID-to-OOD drop of only 8.3% versus 19.0% for IC-E2E. The authors attribute EC-E2E's weaker ID accuracy to possible modality interference from jointly training text generation and action prediction.

• RoboInter-Data exceeds prior datasets in scale and annotation breadth. The comparison table lists RoboInter at 230k videos and 571 scenes, versus Hamster at 136k videos, Robo2VLM at 176k videos and 463 scenes, AgiBot-World at 1M videos and 106 scenes, ShareRobot at 51k videos and 102 scenes, ECoT at 60k videos and 12 scenes, and RH20T-P at 38k videos and 7 scenes. RoboInter is listed as the only entry with dense, embodied-VQA, E2E-action-aligned, and curated multi-intermediate chain-of-thought support simultaneously, using Human+Auto annotation.

Methodology in Plain English

The team started from existing robot data rather than collecting everything from scratch. They gathered raw manipulation video from two kinds of settings — an "In-the-Wild" indoor/scene-diverse group (mainly Droid and OXE) and a "Table-Top" group emphasizing skill quality (RH20T) — screened and preprocessed it, and assembled 230k episodes.

Annotation runs through RoboInter-Tool. Humans decompose each video into clips labeled with one of 15 predefined primitive skills and write clip- and video-level language descriptions (with ChatGPT producing preliminary reference text). They mark the contact frame where the arm touches the object. The tool then hands the object to SAM2 for segmentation and tracking, whose output is returned asynchronously for human review. Because many raw recordings lack reliable camera parameters, the team estimates a calibration matrix, uses gripper detection and point tracking to fill gaps, and reconstructs the end-effector's 2D trace. From these base labels they derive additional representations: grasp affordance boxes and contact points from the contact frame, placement proposals from the object's position at the end of a subtask, and gripper bounding boxes projected from 3D to 2D.

Those annotations are then converted into question-answer pairs across two axes: spatial versus temporal, and understanding versus generation. Spatial tasks include selecting correct boxes or grasp poses, matching scenes to instructions, judging contact, and generating boxes, poses, placements, key points, and gripper boxes. Temporal tasks cover movement direction selection, trace-to-description matching, subtask discrimination, execution-stage identification, success/feasibility judgments, trace generation (under easy and challenging settings with or without initial waypoints), and multi-step planning under varying amounts of prior context.

On top of this data, the models follow plan-then-execute. The Planner is a VLM (from the Qwen-VL series or LLaVA-One-Vision) trained with cross-entropy on the VQA data, producing an F-CoT sequence of intermediate representations in text or visual form. The Executor builds on a Qwen2.5-VL backbone with a Diffusion Transformer action head and an information aggregator that compresses hidden states and intermediate representations into conditioning features of controllable length. It takes multi-view observations and instructions and outputs multi-step action chunks via a diffusion loss. Three paradigms are compared: IC-E2E (inject the pretrained Planner VLM as a stronger feature extractor), EC-E2E (jointly optimize intermediate-representation reasoning and action generation), and Modular (Planner and Executor as independent modules).

Evaluation uses two protocols. Open-loop scoring measures per-step action accuracy against ground truth, averaged over 100K transitions from evaluation videos, with roughly 10% of episodes from each subset used for Executor training (25k total, 8% reserved for evaluation). Closed-loop testing runs four real tasks on a Franka Research 3 arm in a few-shot Table-Top setup.

Why This Matters

The paper's central claim is that intermediate representations are the practical bridge between what a VLM can reason about and what a robot arm must actually do — and that the field has been held back mainly by a shortage of large, well-aligned annotated data rather than by model architecture. By releasing tooling, data, benchmark, and models together, it gives future work one common base to measure against, and it shows that representation granularity matters: dense, spatially and temporally grounded labels (especially traces) help far more than coarse stage-level ones.

Potential real-world applications named or implied in the paper:

• Household and tabletop manipulation, such as collecting objects into a cup, stacking cups, folding towels, and cleaning clutter off a board — the four tasks tested in the real-world experiments.
• Shared-autonomy and human-robot interaction, where traces, bounding boxes, and subtask sequences give human operators an intuitive, precise way to specify what a robot should do.
• Controllable video generation and embodied world models, using the temporally aligned annotations over 640×360 raw videos as structured control signals and supervision.
• Training video action models that need multi-level supervision combining actions, videos, and intermediate representations.

Industry relevance: the resource lowers the annotation cost for anyone building manipulation policies, provides an off-the-shelf benchmark for comparing embodied VLMs, and demonstrates that even small (3B and 7B) pretrained planners can materially improve downstream execution. That matters for teams that cannot afford to pretrain from scratch but can fine-tune a Planner and reuse it across Executor variants.

Future Directions

• Extending beyond the current source datasets. The data is drawn mainly from Droid and RH20T (plus OXE), and the real-world validation is limited to a few-shot TableTop setup on one arm — whether the annotations transfer to other embodiments, laboratories, and task families is left open.

• Scaling and refining annotation automation. The paper notes that purely automated pipelines are sensitive to distribution shift and that prior work with manual verification was small or misaligned with step-wise actions. How far semi-automatic, human-in-the-loop annotation can be pushed toward full automation without losing per-frame alignment is unresolved.

• Training specialized generative models per representation. The authors propose using the dataset's large, high-quality per-representation annotations to pretrain expert generative models tailored to individual intermediates such as traces or affordance boxes.

• Resolving the open-loop versus closed-loop inconsistency. EC-E2E won on open-loop and OOD metrics while IC-E2E won on ID accuracy, and the authors partly explain this by noting that the open-loop protocol enforces strict decoupling and therefore behaves like an OOD test. Whether that explanation is complete, and how to design variants that win on both axes, is an open question.

Target Audience

Robotics and embodied-AI researchers working on VLA models, VLM planners, or plan-then-execute architectures will get the most from this paper, particularly those who need large annotated manipulation data or a ready-made embodied VQA benchmark. It is also useful for dataset builders and annotation-tool developers interested in semi-automatic labeling pipelines, and for practitioners who want to fine-tune a small multimodal model into a manipulation Planner without paying for large-scale data collection. Readers without background in robot learning will find the dataset and benchmark sections accessible, but will need to consult the appendices for the details of the model variants and evaluation protocols, which are only summarized in the main text.

Authors’ abstract

Advances in large vision-language models (VLMs) have stimulated growing interest in vision-language-action (VLA) systems for robot manipulation. However, existing manipulation datasets remain costly to curate, highly embodiment-specific, and insufficient in coverage and diversity, thereby hindering the generalization of VLA models. Recent approaches attempt to mitigate these limitations via a plan-then-execute paradigm, where high-level plans (e.g., subtasks, trace) are first generated and subsequently translated into low-level actions, but they critically rely on extra intermediate supervision, which is largely absent from existing datasets. To bridge this gap, we introduce the RoboInter Manipulation Suite, a unified resource including data, benchmarks, and models of intermediate representations for manipulation. It comprises RoboInter-Tool, a lightweight GUI that enables semi-automatic annotation of diverse representations, and RoboInter-Data, a large-scale dataset containing over 230k episodes across 571 diverse scenes, which provides dense per-frame annotations over more than 10 categories of intermediate representations, substantially exceeding prior work in scale and annotation quality. Building upon this foundation, RoboInter-VQA introduces 9 spatial and 20 temporal embodied VQA categories to systematically benchmark and enhance the embodied reasoning capabilities of VLMs. Meanwhile, RoboInter-VLA offers an integrated plan-then-execute framework, supporting modular and end-to-end VLA variants that bridge high-level planning with low-level execution via intermediate supervision. In total, RoboInter establishes a practical foundation for advancing robust and generalizable robotic learning via fine-grained and diverse intermediate representations.