Differentiable Inverse Graphics for Zero-shot Scene Reconstruction and Robot Grasping

AI summary

Overview

Research area: Robotics and computer vision, specifically zero-shot 3D scene reconstruction, model-free 6D pose estimation, and robotic grasping.

Technical level: Advanced. The paper assumes familiarity with differentiable rendering, inverse graphics, constrained optimization, and 6D pose estimation benchmarks.

Scope: The paper presents a differentiable neuro-graphics model that reconstructs physically consistent 3D scenes from a single RGBD image and bounding boxes, with no 3D training data or test-time samples, and validates the reconstructions on few-shot pose estimation benchmarks and a real-robot grasping task.

What This Paper Is About

Robots operating in unfamiliar environments need to estimate the shape, pose, material, and lighting of objects they have never seen before, and then grasp them. Most current state-of-the-art systems solve this with large training datasets and additional test-time data, producing black-box scene representations that are data-hungry and hard to interpret. This paper instead combines a segmentation foundation model with a physics-based differentiable renderer to solve a cascade of constrained optimization problems, recovering meshes, 6D poses, materials, lighting, and masks from a single RGBD image plus bounding box prompts.

Key Contributions

1. Zero-shot reconstruction pipeline: A multi-stage optimization pipeline that replaces training entirely, reconstructing unseen objects from one RGBD image without any 3D training data. The coarse-to-fine strategy surpasses model-free few-shot pose estimation models on standard 6D pose benchmarks and is validated on a zero-shot robotic grasping task.

2. Robust optimization methodology: A new probabilistic ellipsoid estimation stage using physically consistent priors (Laplace, Log-normal, truncated normal distributions) to account for sensor noise and mask over-segmentation. This serves as a robust initialization for the subsequent Quasi-Newton (L-BFGS) optimization and helps the pipeline avoid local minima, which the authors validate experimentally.

3. Differentiable graphics engine: A novel differentiable ray tracer for triangular meshes built using only JAX primitives, enabling GPU-based optimization with high portability. It includes a new differentiable soft-mask function that operates directly on rendered depth images, addressing the zero-gradient problem inherent in rendering binary masks without requiring pixel-mesh correspondence.

4. A new evaluation dataset: CLEVR-POSE, a CLEVR-based dataset with 50 scenes, each containing one to five unique object instances with 6D pose labels, combining various colors, shapes (sphere, cube, cylinder), sizes (3 cm, 5 cm, 7 cm), and materials (rubber, metal) to test generalization under limited data.

Main Findings

• Ellipsoid initialization is critical: In 500 runs on an RTX 5090 GPU on CLEVR-POSE, the ellipsoid initialization from Eq. 11 achieved 0.656 ± 0.05 AR_VSD on its own, while a baseline using Normal and Truncated Normal distributions fit to dataset statistics failed to converge (0.008 ± 0.04 AR_VSD).

• The 3D line constraint gives only minor stabilization: 0.653 ± 0.06 AR_VSD with the constraint versus 0.648 ± 0.05 without it.

• Stage timings: The ellipsoid and scene optimization stages take 2.76 ± 1.01 s and 4.73 ± 0.31 s respectively. The scene optimization converges in under 300 steps, a 2.3x reduction in optimization steps compared to other methods applied to analogous scenes. Mesh optimization runs for 4K steps using ADAMW with a learning rate of 5e-3, and its time scales linearly with the number of objects at every resolution.

• Renderer benchmarks: Rendering a 512×512 image of a sphere with 162 vertices, the JAX-sphere renderer takes 0.12 ms, 32x faster than PyTorch3D (3.9 ms) and 239x faster than Mitsuba (29.1 ms). JAX-mesh takes 7.8 ms, which is 3.7x faster than Mitsuba and 2x slower than PyTorch3D. A 15K-vertex scene renders a 512×512 image in 0.97 s.

• FewSOL results: Evaluated on 20 objects, DNG variants consistently outperform Gen6D and OnePose++. DNG2, which processes images at 20% of the original 640×480 resolution (128×96), reports Chamfer distances (×10³) ranging from 0.22 to 2.33 across the objects, against Gen6D values ranging from 1.81 to 106.89 and OnePose++ values from 2.17 to 68.31. Hausdorff distances for DNG2 range from 0.040 to 0.239. AR_VSD was omitted for OnePose++ because of substantial mask prediction inaccuracies. Accuracy increases with object size and input resolution.

• MOPED results: DNG2 outperforms Gen6D and OnePose++ across most metrics. DNG2 AR_VSD values per object are 0.656, 0.286, 0.735, 0.450, 0.533, and 0.136, compared with Gen6D's 0.145, 0.004, 0.110, 0.056, 0.143, and 0.073. DNG2 Chamfer distances (×10³) are 1.38, 2.64, 0.49, 1.41, 0.52, and 2.72, against 8.7 to 23.8 for Gen6D and 06.2 to 49.1 for OnePose++.

• LINEMOD-OCCLUDED results: The method achieves a mean AR_VSD of 0.275 ± 0.083 and ranks among the top 100 entries on the LINEMOD-OCCLUDED benchmark of the BOP Challenge leaderboard, despite competing against fully supervised models trained on large real and synthetic datasets with access to 3D object models. SAM achieved a mean IoU of 0.712 (± 0.072) between predicted and ground-truth masks under occlusion.

• Zero-shot grasping: 224 grasping tests on 10 diverse YCB objects yielded a total success rate of 89.28% (200 of 224 attempts). Per-object accuracy ranged from 100% (ball, apple, soup) down to 75% (drill). The comparison method GraspSAM reaches a comparable 86% success rate on 20 similar household objects in cluttered scenes, but was pre-trained on roughly 540 million labeled grasps, whereas this method uses no grasp-specific training data.

• Stated limitations: Total optimization time is approximately one minute with the baseline DNG2 on an RTX 5090 GPU, and the pipeline relies on bounding boxes as input.

Methodology in Plain English

The system takes one RGBD image and a set of bounding boxes and turns them into a full 3D scene description through four sequential stages.

First, an object detector supplies the bounding boxes, and the SAM foundation model uses its original pre-trained weights to produce segmentation masks for each object. For the robotic experiments, the authors used EFFICIENTDET-D7 pre-trained on COCO and a custom SSD512 model fine-tuned on VOC objects.

Second, the depth image is projected into a point cloud, split by mask, and each object's point set is fitted with an ellipsoid. Rather than a plain least-squares fit, the authors set up a maximum a posteriori estimation problem using long-tailed priors (Laplace for position, Log-normal for depth size, truncated normal for width and height) so that sensor noise and stray background points do not distort the fit. This is solved with L-BFGS and gives a robust starting position and size for each object.

Third, a differentiable renderer is used to refine the scene with simple spherical shapes, optimizing lighting position and intensity, object positions, scales, and Phong material properties (ambient, diffuse, specular, shininess, color). Physical plausibility is enforced with barrier functions that keep materials within valid Phong boundaries, and the input image, depth, and masks are compared to the rendered outputs using a weighted sum of mean absolute errors.

Fourth, meshes replace the spheres, with deformation driven by a control cage using mean value coordinates. This stage adds three regularizers, a discrete Laplacian smooth mesh loss, a disparity smoothness depth loss, and a new volume loss that keeps the mesh volume close to the spherical volume estimated in the previous stage. The mesh stage is optimized with ADAMW, and the object's pose is assigned by applying PCA to the reconstructed mesh.

The renderer itself is a differentiable ray tracer written entirely with JAX primitives, which makes it portable and easy to integrate with optimization and deep learning libraries. Because rendering binary masks is non-differentiable, the authors introduce a soft-mask function that works on rendered depth values, normalizing them, assigning a constant negative depth to empty pixels, and applying a scaled sigmoid to produce smooth mask values.

Why This Matters

Impact on research: The work argues that both deep learning and classical differentiable rendering impose data requirements that are misaligned with the goal of robots operating from minimal real-world observations. By showing that a physics-grounded optimization cascade can beat trained model-free few-shot pose estimators without any training data or test-time samples, the paper positions differentiable inverse graphics as a competitive alternative to large-scale supervised learning for robot perception. The fully JAX-based renderer and open-source release plans also lower the barrier for researchers who want to prototype physics-based perception pipelines.

Real-world applications:

• Search-and-rescue robotics, where a robot must identify and manipulate debris or tools it has never encountered and cannot be trained on in advance.
• Domestic service robots that need to pick up arbitrary household objects drawn from the YCB object set without pre-scanned models.
• Warehouse and logistics picking, particularly for objects in cluttered and occluded arrangements, which the LINEMOD-OCCLUDED results directly target.
• Manufacturing or inspection tasks where interpretable, physically consistent scene estimates matter more than raw throughput.

Industry relevance: The paper targets two industrial pain points directly: the cost of collecting millions of labeled 3D samples, and the lack of explainability in black-box perception models. The authors argue that hallucinations in high-torque robotic systems carry immediate physical risk, unlike hallucinations in language models that can be corrected after the fact. A pipeline that produces explicit meshes, materials, and lighting is easier to inspect and validate, and it requires no per-object data collection. The approximately one-minute optimization time on an RTX 5090 GPU remains a practical hurdle for deployment.

Future Directions

• Accelerating inference: The stated total optimization time of roughly one minute per scene is a bottleneck; the authors propose learned priors as a route to faster inference.
• Removing the bounding-box dependency: The pipeline currently relies on bounding boxes, and the authors propose integrating foundation models for object detection to remove this requirement.
• Scaling to more complex and cluttered scenes: The LINEMOD-OCCLUDED evaluation uses a subset with all objects present, and improvements under heavy occlusion remain an open area, especially for objects the reconstruction struggles with, such as the drill and cleaner in the grasping experiments.
• Closing the gap between simulated and real grasping performance: The grasping policy used is deliberately simple, sampling orientations and taking the first successful simulated attempt; the authors acknowledge that more complex policies exist, leaving room to improve the transfer from reconstructed scene to executed grasp.

Target Audience

This paper is most valuable to robotics and computer vision researchers working on zero-shot perception, model-free 6D pose estimation, differentiable rendering, or robot manipulation. It also suits engineers building grasping or pick-and-place systems who want to avoid large labeled 3D datasets, and graphics researchers interested in JAX-based differentiable ray tracing and soft-mask formulations. Readers lacking background in constrained optimization or physically based rendering will find the methods sections demanding.

Authors’ abstract

Operating effectively in novel real-world environments requires robotic systems to estimate and interact with previously unseen objects. Current state-of-the-art models address this challenge by using large amounts of training data and test-time samples to build black-box scene representations. In this work, we introduce a differentiable neuro-graphics model that combines neural foundation models with physics-based differentiable rendering to perform zero-shot scene reconstruction and robot grasping without relying on any additional 3D data or test-time samples. Our model solves a series of constrained optimization problems to estimate physically consistent scene parameters, such as meshes, lighting conditions, material properties, and 6D poses of previously unseen objects from a single RGBD image and bounding boxes. We evaluated our approach on standard model-free few-shot benchmarks and demonstrated that it outperforms existing algorithms for model-free few-shot pose estimation. Furthermore, we validated the accuracy of our scene reconstructions by applying our algorithm to a zero-shot grasping task. By enabling zero-shot, physically-consistent scene reconstruction and grasping without reliance on extensive datasets or test-time sampling, our approach offers a pathway towards more data efficient, interpretable and generalizable robot autonomy in novel environments.