VersaCamVLA: Camera-Configurable VLA Policies for Robotic Manipulation

AI summary

Overview

• Research area: Computer vision and robot learning — specifically Vision-Language-Action (VLA) models for robotic manipulation, with a focus on camera-configurable visual perception.
• Technical level: Advanced. The paper assumes familiarity with VLA policies, transformer encoders/decoders, Plücker ray embeddings, and flow-matching action regression.
• Scope: The paper proposes VersaCamVLA, a two-stage framework that learns a unified fixed-size "scene-token" interface from an arbitrary set of posed RGB views and injects it into a pretrained VLA (π0.5) so the policy keeps working when camera count or camera pose changes at deployment.

What This Paper Is About

VLA policies are trained with a fixed camera setup, so adding, removing, or moving a camera at deployment changes the visual token distribution and degrades the policy — a real problem because camera placement is often dictated by the robot, the workspace, or the task. VersaCamVLA aims to decouple camera-set representation learning from action learning: it maps any non-empty set of posed RGB views into fixed-size latent scene tokens, then conditions a pretrained VLA on a compact version of those tokens. The framework requires no explicit 3D sensing, no novel-view rendering, and no extra multi-pose demonstrations — only RGB images with camera intrinsics and extrinsics.

Key Contributions

1. Formulates camera-configurable VLA policy learning, where a single policy must operate with a variable number of posed RGB views without assuming fixed camera poses, while keeping a fixed-size visual interface for action prediction.
2. Proposes VersaCamVLA, a two-stage framework that learns a unified scene-token interface via multi-signal target-view prediction (RGB, semantic, and edge signals), then injects the resulting scene tokens into a pretrained VLA as supplementary visual conditioning through a lightweight spatial encoder.
3. Introduces Wrist-Augmented Pose Sampling (WAPS), which turns the natural motion of wrist cameras into free pose diversity for scene-token learning, avoiding the cost of collecting additional multi-pose demonstrations.
4. Validates the approach on simulation and real-robot benchmarks, reporting consistent improvements over prior VLA methods and direct multi-view baselines under changing camera counts and unseen camera poses.

Main Findings

• RoboTwin 2.0 (16 fine-grained bimanual tasks, 100 trials per task per setting, 3-view inputs): VersaCamVLA reaches 52.56% average success under Clean and 23.00% under domain-randomized (DR) conditions, the best average in both settings. For comparison, ACT scores 19.69% / 0.06%, DP 23.69% / 0.00%, DP3 39.94% / 2.56%, OpenVLA-OFT 15.19% / 3.56%, and π0.5 35.88% / 21.63%.
• LIBERO (four suites, 50 trials per task): VersaCamVLA averages 95.9%, above Diffusion Policy (72.4%) and OpenVLA-OFT (94.5%) and on par with the π0.5 baseline (95.8%), with slight gains on LIBERO-Spatial (98.2), LIBERO-Goal (95.0), and LIBERO-Long (92.4); LIBERO-Object is 97.8.
• Camera-pose robustness: Under matched four-view inputs, π0.5 drops 16.0 percentage points from seen to unseen camera poses while VersaCamVLA loses only 0.8 percentage points. On RoboTwin 2.0 Clean the averages are 48.88% to 32.88% for π0.5 versus 53.44% to 52.63% for VersaCamVLA; on the real robot they are 41.67% to 13.33% versus 53.33% to 58.33%.
• Camera-count flexibility: A single VersaCamVLA policy evaluated at 3, 4, 5, and 6 input views on RoboTwin 2.0 Clean scores 52.56%, 53.44%, 53.13%, and 55.63%. π0.5 baselines trained separately at 3 and 4 views score 35.88% and 48.88% at their matched counts. On the real robot, one VersaCamVLA model scores 53.33% at 4 views and 65.00% with one extra test-time camera, versus 31.67% (3 views) and 41.67% (4 views) for π0.5.
• Real-robot advantage at matched views: Under matched 4-view inputs, VersaCamVLA outperforms π0.5 by 11.7 percentage points on seen poses and 45.0 percentage points on unseen poses.
• Token count matters: Compressing to 192 scene tokens gives 55.63% average success, versus 50.06% for 48 tokens and 47.50% for 432 tokens.
• Convolutional spatial encoder beats pooling: 55.63% versus 50.13% average success.
• WAPS helps under unseen poses: 52.63% with WAPS versus 48.79% without, a gain of 3.84 percentage points.
• LVSM pretraining initialization helps: Initializing the scene encoder–decoder from a pretrained LVSM checkpoint gives 55.63% versus 51.25% trained from scratch.
• Multi-signal supervision is additive: On RoboTwin 2.0 DR, RGB-only supervision gives 20.94%; adding edge gives 24.75%; adding semantic gives 25.69%; combining both gives 29.50%, an 8.56 percentage-point improvement over RGB-only.

Methodology in Plain English

The design splits the problem into "how do we describe a scene" and "how do we act on it," and trains these separately.

Stage I — learning a scene description. A scene encoder takes any non-empty set of RGB images that come with known camera intrinsics and extrinsics. Each image patch is paired with a Plücker ray embedding that encodes which direction that patch looks along, and the patches from all views are flattened into one variable-length sequence. A set of learnable scene tokens is concatenated to this sequence and processed by a transformer, so the scene tokens can attend to every input view; only the updated scene tokens are kept, giving a fixed-size representation no matter how many cameras were used. A decoder used only during training queries these tokens with a target camera's rays and tries to predict RGB, semantic, and edge maps at that target view. Because the decoder is asked to render views that were not given to the encoder, the scene tokens are pushed to hold view-consistent spatial information rather than just memorizing pixels. Semantic targets come from the simulator's segmentation sensor in simulation and from a SAM3 segmenter fine-tuned on the authors' real-robot data for real frames; edges come from a Canny detector.

Wrist-Augmented Pose Sampling. Learning this interface needs diverse camera placements, which are expensive to collect. WAPS instead exploits the fact that wrist cameras sweep a rich distribution of poses as the robot moves. For each training frame, the static-view subset is sampled by choosing a random integer n between 1 and the number of static cameras, and each wrist view is independently included with probability 0.5. The target set is always the full camera pool, so the decoder reconstructs every available view regardless of what the encoder saw. This yields both self-reconstruction on source views and cross-view prediction on held-out views, all supervised by the same scene tokens.

Stage II — plugging into a policy. The decoder and prediction heads are thrown away, the scene encoder is frozen, and a randomly initialized convolutional spatial encoder compresses the scene tokens into a smaller supplementary conditioning signal. The VLA backbone (π0.5) keeps its native fixed-view RGB input and action-learning objective, and the compressed scene representation is fed in alongside language, RGB, and proprioception. The spatial encoder and the backbone's trainable parameters are optimized jointly with the standard flow-matching action regression loss — it acts as a task-driven information bottleneck because it is supervised only through that loss.

Training and evaluation setup. Training uses a predictive horizon of 50, 30,000 iterations, AdamW with a learning rate of 5×10⁻⁵, 8 NVIDIA H100 GPUs, and a total batch size of 512. Robustness experiments perturb camera azimuth, distance, and pitch over a workspace-centered spherical region. RoboTwin 2.0 uses the Aloha-AgileX embodiment with 6 environment cameras and 2 wrist cameras per frame, 16 tasks selected from the 50-task suite, 50 teleoperated demonstrations per task under Clean, and 100 trials per task per setting. LIBERO uses a Franka Emika Panda arm in MuJoCo with 6 environment cameras and 1 wrist camera, the benchmark's 50 demonstrations per task, and 500 trials per suite. Real-robot experiments use a Cobot Magic (Mobile Aloha) platform with 4 environment cameras and 2 wrist cameras, 50 human-teleoperated demonstrations per task, and 20 evaluation trajectories per task per condition.

Why This Matters

Impact on research. The paper reframes camera sensitivity in VLA policies as a representation-interface problem rather than a data-collection or 3D-reconstruction problem. By showing that a fixed-size scene-token bottleneck can absorb a variable camera set, it offers an alternative to the common practice of concatenating per-camera tokens — an approach the authors note still binds the policy to the training view count and grows the token length with the number of views. The two-stage separation also means a base VLA backbone need not be redesigned.

Real-world applications:

• Deploying the same manipulation policy across robot platforms whose camera mounts differ or whose views are partially occluded.
• Adding a temporary extra camera during evaluation to improve coverage without retraining the policy — the paper demonstrates a single model going from 53.33% at 4 views to 65.00% with one extra test-time camera.
• Fine-grained tasks such as handover-and-insertion, bimanual assembly, and precise insertion where spatial perception is critical and a useful training view may be unavailable on a new platform.
• Reusing existing single-arm and bimanual demonstration datasets (LIBERO, RoboTwin 2.0, and a Cobot Magic setup) without collecting multi-pose data, since wrist-camera motion supplies pose diversity for free.

Industry relevance. The work targets the deployment barrier created by fixed camera setups: a policy that tolerates camera changes reduces the need for per-site retraining and re-collection, which is a practical cost for anyone deploying learned manipulation policies. The requirement of calibrated intrinsics and extrinsics, however, is an engineering precondition rather than something the method removes.

Future Directions

• Uncalibrated cameras. The method assumes known camera intrinsics and extrinsics; extending the scene-token interface to cameras without calibration is left open.
• Active perception. The authors note the framework currently relies on manual camera placement and lacks autonomous viewpoint selection, and suggest optimizing camera configurations online as future work.
• Beyond tabletop setups. The current scope is tabletop bimanual manipulation; mobile workspaces remain an open extension.
• Safety validation. The paper states that safety-critical deployment would require further validation against task-specific failures under large camera-configuration shifts.

Target Audience

This paper is most useful to robotics and embodied-AI researchers working on VLA policies, imitation learning, and multi-camera manipulation; to engineers deploying learned manipulation policies across robot platforms where camera mounts differ; and to readers already familiar with transformer-based vision encoders, camera geometry (intrinsics, extrinsics, Plücker rays), and flow-matching action heads. Because the writing assumes that background, it is not a beginner-friendly introduction.

Authors’ abstract

Vision-Language-Action (VLA) models have emerged as powerful foundations for robotic manipulation, but their reliance on fixed camera configurations during training makes them brittle to changes in camera count or pose during deployment. To overcome these limitations, we propose VersaCamVLA, a camera-configurable framework that decouples camera-set representation from action learning. VersaCamVLA learns a unified scene-token interface that maps an arbitrary, variable set of posed RGB views into fixed-size latent scene tokens. This is achieved via multi-signal target-view prediction and Wrist-Augmented Pose Sampling (WAPS), which leverages natural wrist-camera motion for free pose diversity. At deployment, a lightweight spatial encoder injects these compact scene tokens into a pretrained base VLA as a supplementary visual condition, requiring no explicit 3D sensing or novel-view rendering. Experiments on RoboTwin, LIBERO, and a real-robot platform demonstrate that VersaCamVLA consistently outperforms prior VLA methods and direct multi-view baselines, maintaining robust performance across varying camera counts and unseen camera poses.