UniRAP: Towards Unified Part-level Physical Affordance Reasoning and Actionable Perception

AI summary

Overview

Research area: Computer vision and vision-language perception, specifically part-level affordance grounding and language-guided robotic manipulation (embodied AI).

Technical level: Advanced. The paper assumes familiarity with LISA-style "embedding-as-mask" segmentation, SAM2 memory-based video segmentation, DETR-style detection heads, and 4-DoF grasp parameterization.

Scope: UniRAP is a 3B-parameter unified model built on Qwen2.5-VL and SAM2 that maps text (and optional point/box prompts) to bounding boxes, part-level affordance masks, and 4-DoF grasp poses in both images and videos.

What This Paper Is About

Vision-language models can describe what an object is and roughly where it is, but they cannot say how to physically contact it — which fingertip-sized region to touch, and at what position, orientation, and width. UniRAP closes that gap by treating affordance reasoning as a conditional multimodal generation problem: a language model emits special interface tokens (<SEG>, and <REF> when visual prompts are used), whose hidden states are projected into a dense decoder that simultaneously outputs a segmentation mask, a bounding box, and a 4-DoF grasp pose. The goal is one model that handles general referring segmentation, affordance segmentation, and actionable interaction prediction across images and video without separate pipelines.

Key Contributions

1. A unified multimodal-to-contact-geometry model. UniRAP maps text instructions, image/video input, and optional visual reference prompts (points or boxes) to consistent multi-granularity outputs — object boxes, part-level affordance masks, and 4-DoF interaction poses — in both image and video domains. The authors state it is the first end-to-end, reasoning-aware unified model for actionable affordance perception spanning images and videos.

2. A unified prompting and decoding pipeline. A structured <REF> token folds explicit visual prompts and implicit textual references into a shared <SEG> representation space, so tasks without language annotations can be learned through visual prompts. The Unified Affordance Decoder (UAD) reuses intermediate segmentation features for detection and grasp prediction, sharing representations across tasks of different granularity while preserving general segmentation ability.

3. A curriculum-based transfer training strategy. Four progressive stages — general pre-training, affordance pre-training, visual referring pre-training, and actionable affordance fine-tuning — adapt the model from general visual parsing to interaction-aware perception, addressing data imbalance (abundant segmentation data, scarce grasp annotations) and cross-task interference.

4. Broad empirical validation, including real robot experiments. State-of-the-art or competitive results on referring expression segmentation, referring/reasoning video object segmentation, referring and reasoning affordance segmentation, and the RealVLG-Benchmark, plus a 13-object grasping study on a Franka FR3 arm.

Main Findings

• Strong referring expression segmentation at 3B scale. On RefCOCO/+/g (cIoU), UniRAP reaches 80.9 / 82.6 / 78.1 on RefCOCO val / testA / testB, 75.1 / 79.3 / 70.4 on RefCOCO+, and 76.1 / 77.5 on RefCOCOg val(U) / test(U) — competitive with 7B–8B models such as UniPixel-7B and Sa2VA-8B.

• Reasoning segmentation is competitive but not best. On ReasonSeg, UniRAP obtains 57.9 gIoU and 59.0 cIoU, below VideoLISA (61.4 gIoU, 67.1 cIoU), which the authors attribute to ReasonSeg demanding stronger multi-step reasoning than UniRAP targets.

• Solid video segmentation on ReVOS. UniRAP (3B) achieves 64.1 J&F on the referring split, 59.6 on the reasoning split, and 61.9 overall J&F, outperforming 3B-scale HyperSeg (58.5 referring J&F) and InstructSeg (57.0 referring J&F) while remaining below UniPixel-7B (65.8 referring J&F).

• Best referring affordance segmentation among compared methods. On HANDAL, UniRAP scores 63.9 gIoU / 65.7 cIoU versus RAGNet's 60.5 / 60.3; on GraspNet seen, 70.6 / 64.1 versus 63.3 / 64.0; on GraspNet novel, 51.6 / 42.3 versus 45.6 / 33.2.

• Large zero-shot reasoning gains. On the zero-shot 3DOI set for reasoning affordance segmentation, UniRAP reaches 55.7 gIoU and 54.5 cIoU, improving over RAGNet by 17.6 and 15.1 points respectively (RAGNet: 38.1 / 39.4).

• Grasp prediction is the standout advantage on RealVLG-Benchmark. UniRAP's 4-DoF grasp gAcc is 56.7 (seen), 56.7 (similar), and 63.3 (novel), versus 32.8 / 30.2 / 18.3 for RealVLG-R1 (GSPO, 7B) and 18.9 for the 3B version — described as up to a threefold improvement in the novel setting. RealVLG-R1 retains an edge on bounding boxes (e.g., 89.0 / 89.6 seen for the 7B GSPO variant versus UniRAP's 85.5 / 86.1), which the authors credit to its foundation model's large-scale detection pre-training.

• Each curriculum stage contributes. Affordance pre-training raises 3DOI gIoU by 9.9 (referring, 40.5 to 50.4) and 15.4 (reasoning, 37.5 to 52.9). Visual referring pre-training raises grasp accuracy from 26.4 to 63.3 when combined with the proposed Grasp Head. Actionable affordance fine-tuning raises grasp accuracy from 19.6 to 63.3.

• The regression grasp head is decisive. Swapping the proposed head for a heatmap-based alternative collapses grasp accuracy from 63.3 to 13.4.

• Real-world grasping: 88.5% average success. Across 13 household objects on a Franka FR3 with an eye-in-hand RealSense D435i, UniRAP (4-DoF) achieves 88.5% average success, versus 62.3% for RealVLG-R1 (4-DoF), 55.4% for UniRAP + GraspNet, and 43.8% for RAGNet + GraspNet. The authors note GraspNet-based pipelines remain sensitive to small objects, reflective surfaces, and unreliable depth.

Methodology in Plain English

The system chains a language model to a segmentation decoder, with special tokens acting as the bridge.

1. Text side. A Qwen2.5-VL backbone reads the image or video and the user's instruction. If the instruction contains a <REF> token, a visual prompt (a point or box) is encoded — using random Fourier mapping for coordinates and time, then a linear projection — and injected into the text stream. The model then writes an answer containing a <SEG> token.

2. Feature side. SAM2 encodes the image or video into dense visual features. For video, a memory bank and memory attention carry state across frames so the same object stays tracked consistently.

3. The handshake. The hidden state at the <SEG> position is passed through a small MLP projector to become a prompt embedding for the decoder. In effect, the language model says "this is the thing" and the projector translates that into the decoder's language.

4. The decoder (UAD). A Two-way Transformer combines the visual features, positional encodings, and several task tokens (mask, box, grasp, IoU, object pointer). Three heads then act: a Mask Head upsamples features and modulates them with the mask token to produce four candidate masks, selecting the best by a stability criterion; a Bbox Head starts from the bounding box implied by the chosen mask and predicts a residual offset (FiLM modulation, mask-guided pooling, residual MLP); a Grasp Head samples candidate centers inside the mask by farthest point sampling, encodes each as a normalized offset relative to the mask centroid scaled by box size, and regresses center offsets, orientation (as cos 2θ, sin 2θ), width, and a confidence score per candidate.

5. Training recipe. Losses combine a language-modeling loss with focal and Dice losses for masks, MAE for IoU, cross-entropy for objectness, L1 and GIoU for boxes, and Huber and BCE for grasps, plus a mask-in-box containment loss and a grasp-center/mask consistency loss. Training proceeds through the four curriculum stages, so the model learns general segmentation first, then affordances, then prompt-based interaction cues, then joint fine-tuning on scarce vision-language grasp data.

Why This Matters

Impact on research. The paper reframes affordance perception as a generation-plus-decoding problem rather than a static embedding-alignment problem, and shows that a segmentation decoder's intermediate features can carry grasp geometry as well as masks. It also argues against the text-as-output approach used by RealVLG-R1: emitting grasp coordinates as text is shown to be markedly less precise than regressing them from dense features (63.3 versus 18.3 novel gAcc). That is a concrete, falsifiable claim about architecture choice for embodied perception.

Real-world applications:

• Household and warehouse robot grasping, where a spoken or written instruction ("grab the mug handle") must become a stable contact pose rather than just a bounding box.
• Assistive robotics, where a user indicates an object by pointing (via the <REF> mechanism) rather than describing it precisely.
• Video-guided manipulation, where the memory-based design supports tracking a target through a dynamic scene before committing to a grasp.
• Tool use and articulated objects, where the functional part (handle, blade, knob) differs from the object's geometric center — exactly the part-level distinction the affordance masks capture.

Industry relevance. A 3B model with reported performance near 7B–8B generalists is attractive for deployment on robot-mounted compute; the elimination of a separate GraspNet stage removes a failure-prone geometry pipeline (the paper's own 55.4% versus 88.5% comparison makes this case directly). Companies building picking, bin-manipulation, or service-robot systems have an obvious stake, as do groups building VLM-based robot stacks that currently bolt segmentation and grasp modules together post hoc.

Future Directions

• Reasoning-intensive segmentation is unfinished business. UniRAP trails VideoLISA on ReasonSeg (57.9 gIoU / 59.0 cIoU versus 61.4 / 67.1). Whether stronger chain-of-thought training can be added without sacrificing grasp precision is open.

• Bounding-box accuracy lags the specialized baseline. RealVLG-R1's 7B GSPO variant reaches 89.0 / 89.6 gIoU/cIoU on seen objects versus UniRAP's 85.5 / 86.1. The authors attribute this to foundation-model detection pre-training, implying a data-side or pre-training fix.

• The paper flags its own limitations in Appendix D.1 and future work in D.1/D.2, but that appendix text is not included in the provided content, so the specific limitations and societal-impact discussion cannot be summarized here. The grasp representation itself (4-DoF, planar) is a natural constraint to relax toward full 6-DoF manipulation.

• Scaling and data efficiency. Training-set sizes, annotation volumes, and the exact number of grasp candidates (N_g) are referenced but not reported in the available text; how far the curriculum strategy extends as grasp annotations grow is an untested question. The 13-object real-world study is also small — broader object diversity, clutter, and dynamic scenes are the obvious next evaluation.

Target Audience

Researchers and engineers working on embodied AI, vision-language segmentation, and robot manipulation who already understand LISA-style token-to-mask interfaces and grasp pose parameterization. It is most useful to those deciding how to architect a perception stack that must output actionable geometry rather than semantics alone, and to practitioners evaluating whether a compact 3B model can replace a multi-stage segmentation-plus-grasp-detector pipeline in a real robot deployment.

Authors’ abstract

Vision-language perception has achieved impressive progress in aligning natural language with visual observations, yet grounding high-level semantics into part-level physical interaction remains challenging. To address this gap, we propose UniRAP, a unified model for inferring part-level physical affordances and mapping language instructions to actionable geometric representations. UniRAP formulates this problem as a conditional multimodal generation task, integrating visual inputs, textual instructions, and optional prompts into a shared spatiotemporal representation space through a unified interface token mechanism. To predict executable contact geometry, we introduce a Unified Affordance Decoder (UAD), which jointly performs object detection, part-level affordance segmentation, and 4-DoF interaction pose estimation by leveraging intermediate segmentation features. In addition, we propose a curriculum-based transfer training strategy that progressively adapts the model from general visual parsing to interaction-aware perception, improving data efficiency under limited textual annotations. Experiments show that UniRAP achieves state-of-the-art performance on referring expression segmentation, affordance grounding, and interaction pose estimation, while maintaining strong spatiotemporal consistency in dynamic video scenarios. These results demonstrate the effectiveness of UniRAP as a unified perception framework for language-guided physical manipulation.