SaPaVe: Towards Active Perception and Manipulation in Vision-Language-Action Models for Robotics

AI summary

Overview

• Research area: Robotics, specifically Vision-Language-Action (VLA) models, active perception, and robot manipulation.
• Technical level: Advanced. The paper assumes familiarity with VLA architectures, LoRA adapters, action chunking, 3D geometry encoders, and robot control conventions.
• Scope: SaPaVe is an end-to-end framework that trains a robot to move its own head camera to gather task-relevant visual information and then act on what it sees, using a decoupled camera/action design, a 200k-pair camera-movement dataset, and a new simulation benchmark.

What This Paper Is About

Most robot policies are trained on near-optimal, fixed head-camera views, so they fail when a target object is occluded or outside the frame. SaPaVe addresses this by teaching a policy two linked abilities: semantically deciding how to rotate the camera to reveal what matters, and executing manipulation robustly despite the continuously changing viewpoint that results. The authors also build the data and evaluation infrastructure they found missing: a large camera-movement dataset (ActiveViewPose-200K) and the first simulation benchmark for active manipulation (ActiveManip-Bench).

Key Contributions

1. SaPaVe framework: An end-to-end active manipulation framework that uses a bottom-up, two-stage training strategy. Camera movement and manipulation are placed in a decoupled action space rather than a shared one, which the authors argue avoids cross-interference and reduces data demands.
2. ActiveViewPose-200K: A dataset of 200k image-language-camera movement pairs with detailed semantic annotations, built for learning task-oriented camera control. It is constructed semi-automatically from 4k curated 3D assets from Objaverse, 500 diverse scenes, and 3,000 detailed task templates, with instructions generated by GPT-4o and manually refined.
3. ActiveManip-Bench: The first simulation benchmark for evaluating active manipulation, built on NVIDIA Isaac Sim with a G1 humanoid equipped with a pair of Inspire Hands and an active, head-mounted camera. It covers 12 annotated tasks, 100 objects, and 20 distinct scenes, and is designed to be extensible.
4. Empirical results: Reports gains over fixed-view setups, generalist VLMs, and existing VLA models (GR00T-N1 and π₀) in simulation and on real hardware.

Main Findings

• Camera movement can be learned from specialized data, not from generalist VLMs: On ActiveViewPose-200K splits, SaPaVe (Stage 1) scores 85.5 on Val, 89.1 on Test1, and 78.3 on Test2, averaging 84.3. That compares with Gemini-2.5-Pro at 73.3 / 76.5 / 68.2 (average 72.7), Multi-SpatialMLLM at 72.8 / 74.3 / 63.6 (average 70.2), and Qwen2.5-VL-72B at 63.9 / 65.1 / 58.0 (average 62.3). Section 4.2 describes this as an average score 11.6% higher than Gemini-2.5-Pro with only 2B parameters; the abstract and introduction describe the margin as 16%.

• Fixed cameras are a major bottleneck in simulation: On ActiveManip-Bench, a fixed camera averages 36.17% success (74 / 46 / 11 / 52 / 27 / 7 across Unoccluded Pick-and-Place, Occluded Pick-and-Place, Out-of-View Pick-and-Place, Unoccluded Articulated Manipulation, Occluded Articulated Manipulation, Out-of-View Articulated Manipulation). Out-of-View tasks drop by more than 60%.

• Wrist cameras help when the head is fixed, but not when the head is active: Fixed Camera + Wrist Camera averages 52.33%, Active Camera + Wrist Camera averages 73.16%, and SaPaVe with only the active camera averages 74.83% (85 / 78 / 72 / 76 / 70 / 68). Adding wrist cameras to an active head camera gave minimal gain and degraded some metrics. The authors attribute this to noise information and data mismatch, noting ActiveViewPose-200K contains only head-camera movements and that paired head–wrist data is limited to 20k samples used only in Stage 2.

• Directly fine-tuning existing VLAs is insufficient in the real world: On real-world active manipulation, π₀ scores 55 / 45 / 45 / 35 (average 45.00) and GR00T-N1 scores 60 / 55 / 50 / 50 (average 53.75), while SaPaVe scores 90 / 85 / 85 / 80 (average 85.00) across Occluded Pick-and-Place, Out-of-View Pick-and-Place, Occluded Articulated Manipulation, and Out-of-View Articulated Manipulation. The introduction describes this as exceeding π₀ by 40% and GR00T-N1 by 31.25%. On ActiveManip-Bench, Figure 1c reports SaPaVe's average success rate as 75.2%, and the introduction states it surpasses a fixed-view VLA such as GR00T-N1 by an absolute 58% success rate in simulation.

• Generalization holds across unseen objects, lighting, and scenes: In Table 4, original-task success was 90 for Occluded Pick-and-Place, 85 for Out-of-View Pick-and-Place, 85 for Occluded Articulated Manipulation, and 80 for Out-of-View Articulated Manipulation. Performance under Object 1, Object 2, Light 1, Light 2, Scene 1, and Scene 2 variations stayed within a narrow band (roughly 75 to 95) with no reported collapse.

• Every component matters in ablation: Against the full model's 85.00 average, removing Stage 1 gives 53.75, removing Stage 2 gives 66.25, removing the decoupled action head gives 71.25, removing the camera adapter gives 73.75, and removing Universal Spatial Knowledge Injection gives 68.75. Omitting Stage 1 roughly halves the success rate on Out-of-View Articulated Manipulation (45 vs 80), and omitting Universal Spatial Knowledge Injection causes a 15% drop even on the relatively simple Occluded Pick-and-Place task (75 vs 90).

• Large viewpoint shifts are the hard case: In an additional real-world test with head movements exceeding 90 degrees, only SaPaVe succeeded, at a 45% success rate.

Methodology in Plain English

SaPaVe starts from the observation that camera movement is embodiment-agnostic and easier to learn than manipulation, while manipulation data with matching camera movement is scarce. The authors therefore split the action space rather than merging camera movement into the same action vector as arm joints.

The policy takes an RGB image, an optional set of 3D geometric inputs (such as depth maps and camera intrinsics/extrinsics), and a language instruction, and outputs an action chunk over a horizon k. The head camera action is 2-dimensional (relative pitch and yaw per step); the manipulation action is 26-dimensional joint position deltas for the Unitree G1 humanoid platform, which has dual 7-DoF arms and dual 6-DoF Inspire hands.

Three architectural pieces do the work. A Camera Adapter, implemented with LoRA on the VLM and trained on the 200k camera-movement pairs, learns semantic camera control without overwriting the VLM's original weights. Decoupled Action Heads use two separate decoders so camera movement and manipulation are not forced into the same output space. Universal Spatial Knowledge Injection passes 3D geometry through a spatial encoder inherited from a feed-forward 3D geometry model, adds the resulting spatial tokens to the VLM output tokens, and feeds the mixture into the action head during denoising, so the model can accept a range of geometric configurations without retraining or architectural changes.

Training proceeds in two stages. Stage 1 supervises only camera movement on ActiveViewPose-200K, minimizing mean squared error between predicted and ground-truth ego camera movement. Stage 2 freezes the camera adapter and trains the decoupled action head on a mix of ActiveViewPose-200K and active manipulation robot data, using a weighted sum of head and other-action losses. Evaluation uses success rate throughout, with camera predictions counted as correct if they fall within a tolerance of the ground-truth pitch and yaw changes.

Why This Matters

This paper argues that active perception is not an emergent property of large generalist VLMs, and provides evidence: a 2B-parameter model trained on specialized camera-control data beats much larger general-purpose and spatial VLMs on the camera-movement task. It also shows that more sensors are not automatically better — adding wrist cameras to an already-active head camera did not help, because the paired training data was scarce. For the field, the main contributions are a reusable dataset, a reproducible simulation benchmark for a capability that previously had to be tested on physical hardware, and a training recipe that avoids the data cost of jointly collecting camera and manipulation demonstrations.

Real-world applications:

• Home and service robots retrieving items from cabinets, shelves, or cluttered surfaces where the target starts outside the camera frame.
• Warehouse and logistics picking where objects are occluded by other items or placed outside a fixed overhead view.
• Industrial articulated-object interaction, such as manipulating handles, doors, and valves that are awkwardly positioned relative to a fixed camera.
• Humanoid assistants with articulated heads, where "look up, then reach" is a natural and necessary behavior.

Industry relevance: the results target the practical problem that VLA training data is dominated by fixed-viewpoint demonstrations, which makes policies brittle. A decoupled action space plus a large synthetic camera-movement dataset offers a cheaper path to view-invariant policies than collecting real paired camera-and-arm demonstrations, and the ActiveManip-Bench benchmark gives a reproducible way to compare active manipulation methods without replicating identical physical setups.

Future Directions

• Scaling paired head–wrist data: The authors explicitly state that the scarcity of paired head-and-wrist samples (20k, Stage 2 only) limits learning to integrate both views, and that they will scale this data up.
• Integrating wrist cameras without noise: The paper identifies noisy cues and data mismatch as causes of the wrist-camera performance drop, raising the open question of how to combine active head views with additional dynamic viewpoints.
• Extending the benchmark: ActiveManip-Bench is designed to be extensible with new scenes and objects, but the paper does not report results beyond the current 12 tasks, 100 objects, and 20 scenes.
• Generalization under large viewpoint shifts: The additional test with shifts greater than 90 degrees produced only a 45% success rate even for SaPaVe, which the paper presents as validation of the approach but which also marks the remaining performance gap.

Target Audience

Robotics and embodied-AI researchers working on VLA models, active perception, and manipulation policies will benefit most, particularly those concerned with camera control, occlusion handling, and data-efficient training. It is also useful for engineers building humanoid or mobile manipulator systems who need to decide whether to invest in articulated perception hardware and how to train policies that tolerate viewpoint change. Readers new to robotics will find the framing accessible but will need background in VLA architectures and robot control to follow the methodology and benchmark design.

Authors’ abstract

Active perception and manipulation are crucial for robots to interact with complex scenes. Existing methods struggle to unify semantic-driven active perception with robust, viewpoint-invariant execution. We propose SaPaVe, an end-to-end framework that jointly learns these capabilities in a data-efficient manner. Our approach decouples camera and manipulation actions rather than placing them in a shared action space, and follows a bottom-up training strategy: we first train semantic camera control on a large-scale dataset, then jointly optimize both action types using hybrid data. To support this framework, we introduce ActiveViewPose-200K, a dataset of 200k image-language-camera movement pairs for semantic camera movement learning, and a 3D geometry-aware module that improves execution robustness under dynamic viewpoints. We also present ActiveManip-Bench, the first benchmark for evaluating active manipulation beyond fixed-view settings. Extensive experiments in both simulation and real-world environments show that SaPaVe outperforms recent vision-language-action models such as GR00T N1 and \(π_0\), achieving up to 31.25\% higher success rates in real-world tasks. These results show that tightly coupled perception and execution, when trained with decoupled yet coordinated strategies, enable efficient and generalizable active manipulation. Project page: https://lmzpai.github.io/SaPaVe