Any3D-VLA: Enhancing VLA Robustness via Diverse Point Clouds

AI summary

Overview

• Research area: Embodied AI and robot learning, specifically Vision-Language-Action (VLA) models that combine 3D perception (point clouds) with 2D vision for robotic manipulation.
• Technical level: Advanced. The paper assumes familiarity with VLA architectures, transformer-based vision encoders (DINOv2, SigLIP), point cloud encoders, flow-matching action experts, and simulation-based robot training.
• Scope: The paper proposes Any3D-VLA, a modular pipeline that lifts RGB(D) input into point clouds, encodes them with a pretrained point cloud encoder, fuses the 3D features with 2D patch tokens, and trains on a mixture of simulator, sensor, and model-estimated point clouds to improve robustness and sim-to-real transfer.

What This Paper Is About

Most VLA models take only 2D images as visual input, so their spatial understanding is inherited from 2D backbones and they struggle with small objects, viewpoint changes, and occlusions. The authors run a controlled pilot study over five observation spaces and find that explicitly lifting visual input into point clouds and fusing those 3D features with 2D patch tokens beats implicit 3D priors (such as VGGT-based encoders) and RGBD-as-image-channels. Any3D-VLA then packages this finding into a trainable pipeline that also addresses 3D data scarcity and cross-environment domain gaps by mixing point cloud sources during training.

Key Contributions

1. Any3D-VLA, a modular 2D–3D fusion framework. Visual inputs are lifted to point clouds, compressed, encoded by a pretrained point cloud encoder, aligned patch-wise to 2D tokens, and fused as a gated residual correction to the 2D representation, providing a general plug-in for existing VLA backbones.
2. A hybrid point cloud training strategy plus a large-scale RGBD dataset. The dataset is synthesized from the Objaverse LVIS subset (290 categories, 10,680 instances) with depth from both Isaac Sim rendering and model estimation; hybrid training exposes the model to simulator, sensor, and model-estimated point clouds to reduce the cross-environment domain gap.
3. Extensive simulation and real-world evaluation. Zero-shot real-world tests under scale/shape, viewpoint, and appearance-deprived perturbations, plus post-training on new tasks, showing improved performance and robustness even with noisy or scale-biased depth at deployment.

Main Findings

• Point cloud–2D patch fusion wins the pilot study. In simulation (Table 2), this setting reached 61.1% Single-Trial SR, 80.0% Test SR, and 89.5% Grasp SR, versus 45.3/72.6/80.0 for 2D-only, 56.8/76.8/87.4 for RGBD image-plane, 55.8/78.9/85.3 for implicit-depth RGB, and 46.3/78.9/87.4 for implicit-3D RGB. The Single-Trial gap over the second-best result (56.8%) is 4.3%.
• How depth is represented matters more than whether it is given. The authors attribute the limited gains of implicit and reconstruction-based methods to a lack of precise metric alignment and spatial hallucination, and the limits of RGBD-as-channels to flattening 3D topology onto the image plane.
• Real-world zero-shot result. Across four test sets (Standard, Scale & Shape, Viewpoint, Appearance-Deprived) covering 47 distinct objects, 120 trials, and up to three grasp attempts each, Any3D-VLA (Setting 2, DA3) achieved a maximum overall average success rate of 62.5%, outperforming the strongest baseline SpatialVLA at 33.3% by 29.2%.
• Hybrid training usually beats simulator-only training. Under a fixed inference point cloud source, Setting 2 (hybrid) generally exceeded Setting 1 (simulator only), and DA3-estimated inference often beat RealSense inference.
• Post-training on new tasks. With 100 demonstrations per task and 15 trials per task, Any3D-VLA with hybrid point clouds reached 93.3% (Task 1) and 86.7% (Task 2) when inferring with DA3, versus 33.3%/26.7% for π0.5, 33.3%/53.3% for GraspVLA, and 13.3%/6.7% for SpatialVLA.
• Cell-level comparison across training settings (Table 5). Simulation Test SR: 80.0 (simulator-only, simulator points) → 81.1 (hybrid, simulator points); 78.9 → 82.1 with DA3 points. Real-world zero-shot: RealSense 55.0 → 57.5; DA3 60.0 → 62.5. Post-training Task 1: RealSense 73.3 (sensor-only) → 80.0 (hybrid); DA3 80.0 → 93.3.
• Ablation confirms fusion is necessary (Table 6). 3D-only scored 44.2 Single-Trial SR with 91.6 Grasp SR but weak semantic grounding; 2D-only scored 45.3/72.6/80.0; full 2D–3D fusion (3D + DINOv2-L + SigLIP) reached 61.1/80.0/89.5.
• Latency and compression. Raw point clouds of roughly 30k–60k points are reduced to about 3k–8k points via cropping and 3D compression. On a single NVIDIA RTX 3090, inference speed was 3.0 FPS for the 2D GraspVLA baseline, 2.0 FPS with RealSense, 1.7 FPS with Depth Anything 3, 0.5 FPS with UniDepthV2, and 0.3 FPS with MapAnything. Depth Anything 3 was selected for real-world deployment as the accuracy–latency compromise.
• Public benchmarks. On LIBERO (Object, Goal, Long, Spatial) the method improves over GraspVLA by 13.9%; on CALVIN (ABC→D) it increases the average length by 0.71 versus GraspVLA and is consistently slightly better than SpatialVLA overall. Per-task numbers are deferred to the appendix and are not given in the main text.

Methodology in Plain English

The team first ran a controlled comparison. They took the same robot state and instruction and built five different visual pipelines: plain RGB; RGB plus a depth-pretrained encoder; RGB plus a 3D-foundation-model encoder (VGGT); RGB and depth fed as extra image channels into the same encoder; and RGBD lifted into an actual point cloud. To keep the comparison fair, the image encoder was frozen, only the last four layers of the second branch were fine-tuned, the fusion layer was identical across settings, and everything else in the model and training matched.

The winning design became Any3D-VLA. Camera intrinsics convert each valid depth pixel into a 3D point. Because feeding every point is expensive, the workspace is divided into fixed-resolution 3D grid cells, points sharing a cell are aggregated, and one representative point per cell is kept. A pretrained point cloud encoder (Concerto) turns each point plus its color and normal into a feature. Each point is projected back onto the image plane, assigned to a ViT patch, and its features are scatter-mean aggregated into a patch-level 3D token (with a learnable empty token when a patch receives no points). The 3D token is projected to the token dimension, concatenated with the matching 2D token, passed through a small MLP, and added back to the 2D token as a gated residual with a learnable gate initialized to −2.1972.

The base model pairs a Vision-Language Model (InternLM2 1.8B plus a trainable projector) with a conditional flow-matching action expert via a Progressive Action Generation mechanism, following GraspVLA's training paradigm. The VLM head and action expert are jointly optimized; GRIT grounding data supervises bounding-box tokens while the synthetic RGBD data also supervises grasp pose tokens and end-effector trajectories, with no explicit depth reconstruction loss.

Training data is synthesized: object layouts on a 40 cm × 50 cm tabletop, grasps from BoDex, collision-avoidance planning with CuRobo, executability checks in MuJoCo, and rendering in Isaac Sim with randomized lighting, materials, backgrounds, and camera extrinsics from a single viewpoint, using RealSense D435-matched intrinsics. Depth comes either from the simulator pipeline or from model estimation, stored as 256×256×1 float32. Three training settings are compared: simulator-only, hybrid (each trajectory randomly draws a point cloud source with fixed probability p), and sensor-only for real-world post-training.

Why This Matters

Research impact. The paper reframes a common question in 3D VLA research — "should we add depth?" — as "how should depth geometry be represented?" Its controlled pilot study argues that native sparse 3D structure fused with 2D patches is more useful than implicit spatial priors or flattened depth channels, and it shows that diverse point cloud inputs act as effective data augmentation. It also reports strong results without requiring high-precision depth hardware at deployment.

Real-world applications:

• Tabletop manipulation with commodity or no depth hardware, since model-estimated point clouds often matched or exceeded RealSense sensor point clouds at inference.
• Grasping and placing transparent, textureless, or camouflage objects, where color and texture cues are weak and the model must lean on geometry.
• Post-trained tasks with tight placement requirements, such as moving a flower into a vase or a transparent condiment cup into a specific slot of a cup carrier.
• Deployment in cluttered scenes with intra-class size/shape variation and camera viewpoint shifts, relevant to warehouse, retail, and home settings.

Industry relevance. The hybrid training recipe lowers data-collection barriers by making simulator data and model-estimated depth usable together, and the 1.7–2.0 FPS operating rate — helped by action chunking with chunk size 4 and larger per-step motions, roughly 2–3× longer than higher-frequency policies — is presented as feasible for tabletop manipulation. The impact statement notes that more capable robots may affect manual-labor jobs and introduce safety risks, and recommends safety monitoring, human oversight, and risk assessment for real deployment.

Future Directions

• More platforms and harder tasks. The real-world evaluation covers only a single robotic arm and a limited set of objects; the authors suggest extending to additional robot platforms and environments and to more complex, long-horizon tasks.
• Stronger 3D backbones and representation schemes. The current 3D grid compression and feature extraction is intentionally simple to preserve interpretability of the controlled comparisons; richer 3D backbones may yield further gains.
• Exhaustive exploration of depth/point cloud use. The authors state that many strategies for leveraging depth and point clouds exist and were not all tested, and explicitly note their implementation is not the only viable choice.
• Generalizing beyond specific implementations. The broader claim to be tested is whether introducing mature native sparse 3D (compressed point cloud) representations is a generally promising direction, independent of the particular design proposed here.

Target Audience

Robotics and embodied-AI researchers working on VLA models and manipulation policies; 3D perception researchers interested in 2D–3D representation fusion; and engineers building real-world robot systems who need robustness to noisy or scale-biased depth without expensive depth hardware. Readers need working familiarity with transformer vision backbones, point cloud encoders, imitation learning, and flow-matching action generation to follow the architecture and loss details.

Authors’ abstract

Existing Vision-Language-Action (VLA) models typically take 2D images as visual input, which limits their spatial understanding in complex scenes. How can we incorporate 3D information to enhance VLA capabilities? We conduct a pilot study across different observation spaces and visual representations. The results show that explicitly lifting visual input into point clouds yields representations that better complement their corresponding 2D representations. To address the challenges of (1) scarce 3D data and (2) the domain gap induced by cross-environment differences and depth-scale biases, we propose Any3D-VLA. It unifies the simulator, sensor, and model-estimated point clouds within a training pipeline, constructs diverse inputs, and learns domain-agnostic 3D representations that are fused with the corresponding 2D representations. Simulation and real-world experiments demonstrate Any3D-VLA's advantages in improving performance and mitigating the domain gap. Our project homepage is available at https://xianzhefan.github.io/Any3D-VLA.github.io.