Robo3R: Enhancing Robotic Manipulation with Accurate Feed-Forward 3D Reconstruction

AI summary

Overview

• Research area: Robotics — 3D perception and feed-forward 3D reconstruction for robotic manipulation.
• Technical level: Advanced.
• Scope: The paper proposes Robo3R, a feed-forward model that reconstructs metric-scale, fine-grained 3D scene geometry from RGB images and robot states, together with Robo3R-4M, a four-million-frame synthetic training dataset, and evaluates the resulting geometry on reconstruction benchmarks and four real-world manipulation applications.

What This Paper Is About

Robotic manipulation needs reliable 3D geometry, but depth cameras are noisy, material-sensitive, and degrade on transparent or reflective objects, while existing feed-forward reconstruction models lack the precision and metric consistency needed for physical interaction. The authors build a reconstruction model that predicts accurate, metric-scale scene geometry directly from a small number of RGB images plus robot joint states, in real time, and place that geometry in a canonical robot frame. They then show that using this reconstructed geometry as the 3D input to downstream manipulation systems improves results across imitation learning, sim-to-real transfer, grasp synthesis, and collision-free motion planning.

Key Contributions

1. Robo3R-4M dataset and data pipeline. A large-scale synthetic dataset for perception and reconstruction in robotic manipulation scenarios, comprising four million frames over 100,000 scenes, built with diverse assets, extensive randomization, rich modalities, and annotations.
2. Robo3R model. A feed-forward 3D reconstruction model tailored to robotic manipulation that jointly produces high-fidelity depth estimation, precise camera parameter prediction, accurate metric scaling, and a consistent canonical coordinate system, in real time.
3. Specialized architectural components. A masked point head that decomposes dense point prediction into depth, normalized image coordinate, and mask prediction; a relative pose head for multi-view point registration; similarity transformation (S.T.) tokens for global metric scaling; and a keypoint-based PnP extrinsic estimation module that refines the global similarity transformation.
4. Extensive qualitative and quantitative validation. Experiments showing Robo3R is a superior alternative to depth cameras, with higher-quality 3D representation, greater robustness to varying object materials and challenging scenarios, and improved spatial perception for manipulation.

Main Findings

• Point map estimation. On a benchmark of 2,000 scenes and 80,000 frames, Robo3R achieves the lowest errors in both monocular and binocular settings. Monocular: point error 0.006 and scale error 0.007, versus π³ at 0.061/0.497, VGGT at 0.126/0.663, DA3 at 0.075/0.506, MA at 0.078/0.467, and fine-tuned MapAnything (MA-FT) at 0.010/0.010. Binocular: point error 0.005 and scale error 0.004, versus π³ at 0.032/0.483, DA3 at 0.042/0.719, MA at 0.076/0.540, MA-FT at 0.009/0.007, and VGGT at 0.220/0.619.
• Metric scale recovery. The paper reports that other models suffer from severe scale ambiguity with scale errors above 0.46, whereas Robo3R effectively recovers metric geometry. The monocular point error of 0.006 is described as an order of magnitude improvement over the second-best method, π³.
• Relative camera pose prediction. Robo3R achieves RTE 0.014 and RRE 0.013, described as roughly 8 times and 5 times lower than the best baseline π³ (RTE 0.116, RRE 0.073). RTA@0.03 is 0.951 and RRA@0.03 is 0.899, compared with 0.110 and 0.245 for π³, 0.129 and 0.226 for DA3, 0.031 and 0.069 for MA, and 0.014 and 0.052 for VGGT.
• Outperforming a fine-tuned baseline. Robo3R consistently outperforms MapAnything fine-tuned on Robo3R-4M (MA-FT), which the authors attribute to benefits of the proposed model architecture.
• Fine-grained geometry in the real world. Qualitatively, Robo3R reconstructs objects as narrow as 1.5 mm (spanning only 1 to 2 pixels in the image), where other methods and depth cameras fail. It also handles reflective and transparent objects that blind depth sensors, and produces clean point clouds in cluttered scenes containing bimanual robots with dexterous hands, as compared with π³, the depth completion model LingBot-Depth, and a RealSense D455 depth camera.
• Imitation learning gains. With the Maniflow (MF) policy, Robo3R succeeds 14/16 on Sweep Bean and 15/16 on Insert Screw, versus 4/16 and 7/16 for the depth camera, 10/16 and 2/16 for RGB, and 11/16 and 4/16 for π₀. On Breakfast and BiDex Pour, Robo3R scores 12/16 and 16/16 versus 11/16 and 16/16 for the depth camera, 5/16 and 9/16 for RGB, and 4/16 and 12/16 for π₀. The "Other FFs" baseline failed to produce feasible actions (marked "-") due to scale and accuracy errors and inability to crop cluttered background.
• Sim-to-real transfer. Using 200 demonstrations per task collected in NVIDIA Isaac Sim, Robo3R scores 16/16 on Push Cube and 12/16 on Pick Cube, versus 7/16 and 5/16 for the depth camera and 3/16 and 2/16 for RGB.
• Grasp synthesis. With AnyGrasp, Robo3R scores 14/16 on normal objects, 10/16 on transparent or reflective objects, and 11/16 on small objects, versus 12/16, 7/16, and 6/16 for the depth camera. The other feed-forward reconstruction models failed in all three categories.
• Collision-free motion planning. With cuRobo, Robo3R scores 5/5 on normal, 4/5 on transparent or reflective, and 5/5 on thin obstacles, versus 5/5, 2/5, and 2/5 for the depth camera. The other feed-forward models failed in all categories.
• Keypoint + PnP beats direct regression. The extrinsic estimation module lowers absolute translation error from 0.018 to 0.016 and absolute rotation error from 0.018 to 0.016, and raises ATA@0.01 from 0.334 to 0.442 and ARA@0.01 from 0.359 to 0.415 compared with direct extrinsic prediction.
• Robot state conditioning helps. Fusing robot states with image features via element-wise addition gives point error 0.005, normal error 0.079, ATA 0.903, and ARA 0.838, compared with 0.006, 0.081, 0.903, and 0.831 without state and 0.006, 0.086, 0.900, and 0.821 for self-attention-based fusion.
• Masked point head matters qualitatively. A variant retaining the point decoder but removing the masked design fails to recover geometry as fine as 1.5 mm, which the authors say conventional benchmarks do not capture well.

Methodology in Plain English

The model takes in one or two RGB images plus the robot's joint angles and predicts a set of 3D attributes: a depth map, normalized image coordinates, relative camera translation and rotation, and a global similarity transformation.

• Encoding and fusion. Images are encoded with DINOv2 ViT-L into patch features, and robot states are projected into state features with an MLP using GeLU activations. The two are combined by element-wise addition, and learnable similarity transformation tokens are appended.
• Backbone. A transformer backbone using the Alternating-Attention mechanism processes the features through 18 alternating global and frame-wise attention blocks, propagating information both within and across frames.
• Local geometry first, metric scale second. Rather than predicting world-frame metric points directly, the model predicts scale-invariant local points in the camera coordinate system by unprojecting depth along a ray direction. Multi-view points are registered using the predicted relative poses, then mapped into the canonical robot frame by the predicted global similarity transformation.
• Sharp points via masking. Dense point prediction is decomposed into depth, image coordinate, and mask components across three branches — robot, objects, and background — each with a depth head, ray head, and mask head. Region-specific points are extracted by their masks and aggregated, which counters the over-smoothing that blurs point cloud edges.
• Refining camera pose with robot keypoints. The system computes 3D keypoints at each link's frame origin by forward kinematics, projects them during data synthesis into Gaussian-blob heatmaps, and during inference extracts 2D keypoints from predicted heatmaps with a differentiable Soft-Argmax for sub-pixel accuracy. Camera extrinsics are then obtained by solving a Perspective-n-Point problem, which refines the similarity transformation.
• Training data. Robo3R-4M is generated in NVIDIA Isaac Sim using Path Tracing, with physics simulated at 30 Hz (dt ≈ 0.033 s). It uses 16,911 objects from DTC and Objaverse, 4,710 textures, and 6,512 environment maps, with domain randomization over robot behaviors, camera intrinsics and extrinsics, object instances and poses, tables, background, and lighting.
• Training objective. The model is trained end-to-end on a weighted sum of six losses: point loss (L1 over unprojected points after scale alignment), normal loss (angular difference between predicted and ground-truth surface normals), mask loss (binary cross-entropy on robot, object, and background masks), relative pose loss (Huber on translation plus angular error on rotation), similarity transformation loss (Huber on scale and translation plus angular error on rotation), and keypoint loss (L1 on both heatmaps and coordinates).
• Evaluation setup. A benchmark is rendered with objects, textures, and environment maps different from training, containing 2,000 scenes and 80,000 frames, comparing against VGGT, π³, DepthAnything3, MapAnything, and MapAnything fine-tuned on Robo3R-4M. Real-world tests use a single-arm Franka Research 3 with a parallel gripper and a bimanual UR5e with XHands, RealSense D455 cameras, and an NVIDIA RTX 4090, with control at 10 Hz.

Why This Matters

Robo3R reframes 3D sensing for robotics as a learned, calibration-free alternative to depth hardware, and shows that reconstruction quality translates directly into downstream manipulation success rather than being an isolated perception metric. Its emphasis on metric scale, canonical robot-frame geometry, and robustness to transparent, reflective, and tiny objects targets the exact failure modes that limit real deployments.

Real-world applications:

• Grasp synthesis on transparent, reflective, or small objects that depth cameras miss, demonstrated with AnyGrasp.
• Collision-free motion planning around thin or transparent obstacles that depth sensors cannot see, demonstrated with cuRobo.
• Imitation learning for precision and long-horizon tasks such as inserting a screw into a hole only 2 mm wider than the screw, sweeping small beans, toasting bread and pouring milk, and bimanual dexterous pouring.
• Sim-to-real transfer, where a consistent 3D scene representation across simulation and reality reduces the visual gap for policies trained in Isaac Sim.

Industry relevance: The paper argues Robo3R is more cost-effective, accurate, and robust than depth cameras and requires no calibration, which matters for robot platforms where depth hardware is a cost, calibration, and reliability bottleneck. It also shows the model runs on a single NVIDIA RTX 4090-class GPU at a 10 Hz control frequency in the real system.

Future Directions

• Extending beyond pinhole cameras to other camera models, such as fisheye and panoramic cameras.
• Expanding support to a wider range of embodiment types, since the current model covers a limited set.
• Generating new data to fine-tune Robo3R for these new camera models and embodiments.
• Open questions remain about how far the synthetic-to-real transfer demonstrated here extends to unseen scenes, materials, and robot morphologies, and about the reported inference speed trade-offs detailed in the paper's appendix.

Target Audience

Robotics researchers and engineers working on manipulation, 3D perception, or visuomotor policy learning; practitioners building grasp synthesis, motion planning, or imitation learning systems who need better 3D input than depth cameras provide; and machine learning researchers interested in feed-forward 3D reconstruction applied to embodied settings. The paper assumes familiarity with transformer architectures, camera geometry, PnP, and standard manipulation benchmarks, so it is best suited to readers with an intermediate-to-advanced background.

Authors’ abstract

3D spatial perception is fundamental to generalizable robotic manipulation, yet obtaining reliable, high-quality 3D geometry remains challenging. Depth sensors suffer from noise and material sensitivity, while existing reconstruction models lack the precision and metric consistency required for physical interaction. We introduce Robo3R, a feed-forward, manipulation-ready 3D reconstruction model that predicts accurate, metric-scale scene geometry directly from RGB images and robot states in real time. Robo3R jointly infers scale-invariant local geometry and relative camera poses, which are unified into the scene representation in the canonical robot frame via a learned global similarity transformation. To meet the precision demands of manipulation, Robo3R employs a masked point head for sharp, fine-grained point clouds, and a keypoint-based Perspective-n-Point (PnP) formulation to refine camera extrinsics and global alignment. Trained on Robo3R-4M, a curated large-scale synthetic dataset with four million high-fidelity annotated frames, Robo3R consistently outperforms state-of-the-art reconstruction methods and depth sensors. Across downstream tasks including imitation learning, sim-to-real transfer, grasp synthesis, and collision-free motion planning, we observe consistent gains in performance, suggesting the promise of this alternative 3D sensing module for robotic manipulation.