RobotSeg: A Model and Dataset for Segmenting Robots in Image and Video

AI summary

Overview

Research area: Computer vision — image and video object segmentation, specialized for robotic perception.

Technical level: Advanced. The paper assumes familiarity with foundation segmentation models (SAM, SAM 2), transformer attention mechanisms, memory-based video propagation, and standard segmentation metrics such as J&F. The conceptual motivation, however, is accessible to anyone working in robotics.

Scope: The paper introduces RobotSeg, a SAM 2-based foundation model for automatic, structure-aware robot segmentation in images and video, together with VRS, the first large-scale video robot segmentation benchmark.

What This Paper Is About

General-purpose segmentation models like SAM 2 work well on everyday objects but break down on robots, which come in wildly different shapes, blend into cluttered lab backgrounds, have complex articulated joints, and change shape rapidly during manipulation. The paper's goal is to build a dedicated segmentation model that handles these four failure modes, runs automatically without a human clicking on the robot first, and can be trained without the expensive per-frame mask annotations that SAM 2 normally requires.

Key Contributions

1. RobotSeg, the first foundation model for robot segmentation that works on both images and video. It produces fine-grained masks for the robot arm, the gripper, and the whole robot, and still accepts optional user prompts (clicks or boxes) to refine its output.

2. The Video Robot Segmentation (VRS) dataset, the first video-level benchmark for this task: 2,812 videos and 138,707 frames spanning 10 robot embodiments (Franka, UR5, Kuka iiwa, Sawyer, xArm, WindowX, and others), multiple scenes, and varied lighting. It is roughly 38x larger than the existing image-only RoboEngine dataset.

3. Three architectural and training innovations on top of SAM 2: a structure-enhanced memory associator (SEMA) for temporally consistent, structure-aware propagation; a robot prompt generator (RPG) that replaces manual clicks with learned semantic and object tokens; and a label-efficient training strategy (LET) that supervises video learning using only the ground-truth mask from the first frame.

4. State-of-the-art results at a fraction of the parameter cost — 41.3M parameters versus 638.5M or more for competing robot segmentation models and over 13B for language-conditioned alternatives like LISA.

Main Findings

• Automatic segmentation dominates the prior state of the art. On VRS, RobotSeg reaches 85.1 J&F for whole-robot segmentation in fully automatic mode, versus 74.1 for the original RoboEngine model and 38.9 for RoVi-Aug. It beats these baselines by at least 4.9 J&F even when they are given the advantage of being tuned for their task.

• Prompting helps, but the automatic path is already strong. Adding a single click lifts whole-robot performance to 85.1, three clicks to 86.3, and bounding boxes to 85.8 — a modest gain, confirming that the RPG is doing most of the work. Competing promptable models like SAM 2.1 collapse to 38.2 J&F without clicks.

• Structure enhancement is the single biggest architectural win. In the ablation, adding the robot prompt generator raises performance from 77.4 to 83.1 J&F, and the structure-enhanced memory associator pushes it further to 85.1. The structure branch specifically prevents the fragmented, broken-limb masks that plague baselines on articulated robots.

• First-frame-only supervision is viable. With only the first frame annotated, the LET strategy (cycle, semantic, and patch consistency losses) improves the fine-tuned baseline from 73.6 to 77.4 J&F. Each of the three losses contributes incrementally.

• The model generalizes across embodiments. Per-category analysis across all 10 robots shows RobotSeg is the most consistent performer, scoring between 78.7 and 95.6 J&F on whole-robot segmentation. Baselines swing wildly — RoboEngine drops to 6.4 J&F on the Hello Stretch robot, and SAM 3 scores 0.0 on two categories.

• It transfers back to images. On the RoboEngine image benchmark, RobotSeg reaches 87.9 J&F in automatic mode, exceeding even the fine-tuned RoboEngine model's 86.6 while being roughly 20x smaller.

• Mask quality matters for downstream data augmentation. The supplementary material shows that inaccurate masks from RoboEngine and SAM 2.1 produce broken, implausible robot composites when used for augmentation, whereas RobotSeg's clean masks yield structurally faithful synthetic data.

• Training is efficient. The full model trains in 15 hours on 8 RTX A5000 GPUs (24 GB each) for 25 epochs.

Methodology in Plain English

The team started with SAM 2, a strong general-purpose video segmentation model, and fixed the three things that make it ill-suited to robots.

Structure. SAM 2 tracks objects by remembering what they looked like in past frames, but it has no concept of a robot's rigid links and joints. The authors added a branch that runs a Canny edge detector on the current frame, extracts multi-scale features from those edges, and uses attention against the memory to predict a "structure map." That map then modulates the main features, effectively telling the model where robot boundaries are. This keeps limbs attached instead of letting the mask fragment.

Autonomy. SAM 2 needs a human to click on the target. The authors replaced this with a prompt generator that produces two kinds of tokens. Class tokens are pulled from a learned bank based on the requested category (arm, gripper, or whole robot) and supply a semantic prior. Object tokens come from clustering the features inside previously segmented regions using a hierarchical two-level scheme — first splitting the foreground into coarse regions with K-means seeded by farthest-point sampling, then breaking each region into finer subclusters. Together these tokens act as the prompt.

Label efficiency. Normally, training a video segmentation model requires a hand-drawn mask on every frame. Instead, the model predicts forward from frame 0 to frame t and then backward from t to 0, and both endpoints are checked against the single ground-truth mask available (cycle consistency). Intermediate predictions are encouraged to have feature embeddings that match the first-frame object's embedding (semantic consistency), which stops the model from cheating by just holding the mask static. Finally, DINOv3 patch-level feature similarity propagates the first-frame mask into pseudo-labels for intermediate frames, giving a spatially fine-grained training signal (patch consistency).

The result is a compact 41.3M-parameter model that segments robots automatically and consistently across video.

Why This Matters

Robot segmentation is an enabling capability rather than an end in itself. Clean, temporally stable robot masks unlock several downstream systems, and the paper's main practical argument is that sloppy masks (missing parts, drifting boundaries) directly corrupt whatever they feed into.

Real-world applications:

• Visual servoing in vision-language-action (VLA) systems. Precise masks let a policy localize its own effectors in the scene, which supports fine-grained manipulation control.
• Scalable robot-centric data augmentation. A high-quality robot mask lets you cut the robot out and paste it into new backgrounds or swap in a different embodiment, improving policy generalization across robot types and environments without collecting new demonstrations.
• Real-to-sim transfer. Cleaner robot geometry and appearance extracted from real footage produces more accurate simulation assets, narrowing the sim-to-real gap.
• Safety monitoring. Continuously tracking a robot from a third-person view lets a system detect unsafe motions and anticipate collisions with humans in shared workspaces.

Industry relevance: The parameter count is the headline for deployment. At 41.3M parameters, RobotSeg is small enough to run on modest hardware, unlike LISA (13.9B), RoboEngine (898.4M), or RoVi-Aug (638.5M). Combined with its ability to run without human clicks, it is a candidate for always-on perception modules on real robots rather than an offline annotation tool. The semi-supervised training recipe also lowers the annotation cost of adapting to a new robot platform, which is the practical bottleneck for any company deploying many robot types.

Future Directions

• Extending beyond the 10 covered embodiments. The VRS dataset covers common research arms, but industrial and humanoid robots are unrepresented. The category-wise results show the model is not uniformly strong across platforms, so broader embodiment coverage remains open.

• Scaling the semi-supervised recipe. The label-efficient strategy was validated with first-frame-only supervision. It is unclear how far the annotation budget can be pushed down further — for example, using sparse keyframe masks scattered through a video, or no masks at all with purely self-supervised objectives.

• Closing the prompt gap. RobotSeg's automatic mode (85.1) is barely improved by three clicks (86.3), suggesting the prompt generator has nearly saturated what user prompts can add. Whether interactive refinement can deliver larger gains on harder cases — deformable grippers, heavy occlusion, transparent parts — is unresolved.

• Tightening the connection to downstream policies. The paper demonstrates mask quality visually for data augmentation but does not run a closed-loop experiment showing that better masks translate into higher policy success rates. Quantifying that end-to-end benefit is the natural next step.

• Real-time performance. Inference speed is discussed in the supplementary material but not in the main text. For safety monitoring and closed-loop control, latency matters as much as accuracy.

Target Audience

Robotics and embodied-AI researchers building perception pipelines, particularly those working on visual servoing, VLA policies, sim-to-real transfer, or data augmentation for manipulation. Also relevant to computer vision researchers interested in domain-specialized foundation models, semi-supervised video object segmentation, and how far a small model can be pushed by task-specific architectural priors. Practitioners who need to annotate or segment robots at scale — for dataset curation or for production perception — will find the dataset and the 41.3M-parameter model directly usable. Beginners will follow the motivation and results but will need background in segmentation metrics and transformer architectures to engage with the method.

Authors’ abstract

Accurate robot segmentation is a fundamental capability for robotic perception. It enables precise visual servoing for VLA systems, scalable robot-centric data augmentation, accurate real-to-sim transfer, and reliable safety monitoring in dynamic human-robot environments. Despite the strong capabilities of modern segmentation models, surprisingly it remains challenging to segment robots. This is due to robot embodiment diversity, appearance ambiguity, structural complexity, and rapid shape changes. Embracing these challenges, we introduce RobotSeg, a foundation model for robot segmentation in image and video. RobotSeg is built upon the versatile SAM 2 foundation model but addresses its three limitations for robot segmentation, namely the lack of adaptation to articulated robots, reliance on manual prompts, and the need for per-frame training mask annotations, by introducing a structure-enhanced memory associator, a robot prompt generator, and a label-efficient training strategy. These innovations collectively enable a structure-aware, automatic, and label-efficient solution. We further construct the video robot segmentation (VRS) dataset comprising over 2.8k videos (138k frames) with diverse robot embodiments and environments. Extensive experiments demonstrate that RobotSeg achieves state-of-the-art performance on both images and videos, establishing a strong foundation for future advances in robot perception.