GATOR: Generative and Agentic 3D Object Reconstruction From Casual Images

AI summary

Overview

Research area: Computer vision — 3D object reconstruction and generative 3D modeling, with an agentic (LLM-driven) refinement stage.

Technical level: Advanced. The paper assumes familiarity with diffusion/flow-based 3D generative models, camera intrinsics and poses, pointmaps, and differentiable rendering.

Scope: GATOR is a framework that reconstructs complete, textured, pose-aware 3D object assets from one or a few casual images of an object still embedded in its scene, using a generative cascade followed by an agent that edits the result.

What This Paper Is About

Turning ordinary photographs into usable 3D assets is hard because a casual photo shows only part of an object: the back is hidden, other objects occlude it, and lighting and viewpoint vary. A generative model can invent the missing parts, but the invention may not match the real object, and it offers little explicit control over surface regularity, part connections, or texture.

GATOR addresses this by combining a generative 3D model that is conditioned on the images, target masks, depth-derived pointmaps, and text, with a multimodal agent (GPT-6-Astra) that starts from the generated asset and makes targeted geometry and texture repairs. The output is a complete, textured object together with its pose relative to the scene it came from, so objects can be placed back into a shared scene.

Key Contributions

1. Local modality mixing for pose-aware generation. A modality mixer applies attention among patch-aligned RGB, target-mask, and pointmap tokens at each patch within a view, before cross-view attention generates sparse structure. Geometry and appearance are then generated by projecting image features onto that structure, and scene-relative orientation is preserved by pose-aligned training.

2. Text-guided semantic conditioning across the whole cascade. Category names are encoded with frozen SigLIP2 features to modulate structure generation, and captions are encoded with a frozen T5Gemma2 encoder plus a learned adapter to supply tokens for geometry and appearance flows, through stage-specific adapters.

3. Agentic geometry and appearance refinement. The generated, scene-aligned asset initializes GPT-6-Astra, which has Blender access and the input observations, and runs an edit-render-review loop: it compares renders of the original and edited assets under matched cameras and lighting and reverts edits that degrade the reconstruction. No retraining of the generative model is required.

4. Evaluation across synthetic and real casually captured settings. Tests cover Toys4K (525 objects from 105 categories) and four real datasets — LM-O, HB, HANDAL, and ScanNet++ — including clutter, occlusion, lighting variation, and uncertainty in estimated masks, cameras, and depth, plus time-budget comparisons and a scene-level simulation.

Main Findings

• Toys4K, four views: GATOR leads all seven metrics in Table 1, with CD 0.009, NC 0.801, F1 0.757, 2D IoU 0.819, PSNR 17.60, SSIM 0.873, and LPIPS 0.191. Against the strongest baseline, Pixal3D-MV (CD 0.011, F1 0.701), this is a 19.7% reduction in CD and an F1 increase from 0.701 to 0.757.

• Real-world tabletops (LM-O, HB, HANDAL), four views: Excluding the fully proprietary GPT-6-Astra baseline, GATOR leads all nine metrics in Table 2, with mean ADD-SB 0.022, ADD-SB@0.1 of 98.47, CD 0.009, NC 0.859, F1 0.904, 2D IoU 0.841, PSNR 17.97, SSIM 0.890, and LPIPS 0.140. It roughly halves Pixal3D's CD of 0.018. ReconViaGen and Pixal3D lack scene placement, so joint ADD-SB is unavailable for them; ShapeR outputs untextured meshes, so appearance scores are unavailable.

• Indoor scans (ScanNet++): The evaluation protocol is extended from 69 objects in 6 scenes to 229 targets from 49 scenes, with one to ten views each. GATOR leads every metric with ADD-SB mean 0.041, ADD-SB@0.1 of 95.63, CD 0.031, NC 0.726, F1 0.603, and 2D IoU 0.733, reducing CD by 23% relative to the runner-up RecGen and raising F1 from 0.540 to 0.603 and IoU from 0.682 to 0.733.

• Pose recovery: Measured before any object-to-reference alignment, GATOR reaches ADD-SB@0.1 of 96.36% on HANDAL and 95.63% on ScanNet++, with the lowest mean ADD-SB on both the tabletop and room-level benchmarks.

• Ablation (55 HANDAL objects, four views, Table 4): Starting from the base model (mean ADD-SB 0.057, ADD-SB@0.1 94.55, CD 0.010, F1 0.877), the modality mixer lowers mean ADD-SB to 0.037, which the authors attribute to improved robustness to upstream camera and depth estimation errors. Adding text conditioning improves all nine metrics (ADD-SB 0.032, ADD-SB@0.1 98.18, CD 0.008, F1 0.930, PSNR 18.93, LPIPS 0.102). Adding agentic refinement further improves CD to 0.007, NC to 0.829, F1 to 0.936, PSNR to 19.60, and LPIPS to 0.097, while slightly worsening 2D IoU to 0.717 and SSIM to 0.918.

• Efficiency: Without refinement, GATOR takes about 0.5 minutes and already outperforms GPT-6-Astra run with a 20-minute budget; generation is estimated at 32 seconds per object. GPT-6-Astra returns valid assets for 0/10 objects at a one-minute budget, 5/10 at two minutes, and all ten from five minutes onward.

• More views help: A single model supports varying view counts, and going from two to eight views on LM-O and HB lowers ADD-SB and CD. Even with two views, its CD is about half of Pixal3D's with eight views.

• Mask role differs by stage: The structure flow sees the full crop with the mask as separate tokens so it can account for occlusion, while geometry and appearance flows project features from target-masked images so occluders and background do not leak into surface detail or texture — unlike TRELLIS.2, Pixal3D, and ReconViaGen, which condition every stage on background-removed images.

Methodology in Plain English

Each input view supplies an RGB image, a mask marking the target object, camera intrinsics and pose, and a depth map. All of these, except sometimes the mask, can be estimated rather than known. The masked depth is unprojected to 3D to define an "observation anchor" (a center and an isotropic scale) used to normalize coordinates without rotating anything, so outputs keep the scene's orientation. Metric size requires metrically scaled input depth.

Every view is placed on a shared patch grid. At each patch, the model holds three tokens: one from frozen DINOv3 features of the RGB crop, one from the mask, and one from a trainable encoder of the depth-derived pointmap. A "modality mixer" lets these three tokens attend to each other within the patch, then keeps one token per modality. Because the three cues are seen together, the model can cross-check noisy depth against appearance and the mask.

Dense structure latents then query all mixed tokens across all views by cross-attention — used here because the sparse structure is not yet known — while a pooled SigLIP2 embedding of the category name modulates each block. The predicted structure is decoded into sparse occupancy.

Given that structure, image features are lifted to its active 3D locations: DINOv3 features plus their NAF-upsampled versions, sampled from target-masked images and averaged over valid views, with zeros where no view sees a point. Geometry and appearance (PBR material) flows are trained on ground-truth sparse structure and run on the predicted structure at inference, guided by global image tokens and T5Gemma2 caption tokens. A final placement step refines translation and isotropic scale by differentiable rendering of silhouettes and depth and by point alignment, keeping the predicted orientation and shape.

Finally, the complete asset is handed to GPT-6-Astra alongside the input images and Blender. The agent is instructed to make targeted repairs preserving reliable regions and the asset's coordinate frame, writing scripts to complete missing parts, repair topology, or fix UVs and materials. After each edit it renders original and candidate under matched cameras and lighting and reverts harmful edits, stopping when repairs are done or evidence runs out within a fixed time budget (ten minutes per object in the main experiments, with extra-high reasoning effort).

Training uses only synthetic data: procedurally cluttered scenes composed from roughly 980,000 object assets plus structured indoor scenes from existing datasets, with observations augmented or replaced by image-estimated cameras and depth. Evaluation uses the ADD-SB pose-and-shape metric and its recall at 10% of the reference diameter, plus chamfer distance, normal consistency, F1 at 1% (Toys4K) or 2% (real data) of the diameter, 2D silhouette IoU, and PSNR/SSIM/LPIPS on renders under shared lighting. GATOR, Pixal3D, and GPT-6-Astra share Pi3X camera and depth estimates.

Why This Matters

Impact on research. The paper argues that how conditioning signals are fused matters as much as which signals are used: mixing RGB, mask, and pointmap at the patch level before cross-view reasoning, rather than concatenating independent streams, is what produces the pose and geometry gains in the ablation. It also shows a hybrid route — generative model plus agent — where the generative asset grounds the agent to instance-specific geometry and pose, avoiding the cost and shape simplification of building assets from scratch (the paper notes GPT-6-Astra can recover recognizable structure yet reduce parts to simplified shapes). Reconstructed assets are reported to be simulation-ready in a scene-level test.

Real-world applications:

• Creating simulation-ready 3D assets for robotics and embodied AI from phone photos of real objects.
• Scene editing and content creation: reconstructing objects from room scans and recomposing them using their predicted poses.
• Gaming and interactive media, where the paper specifically motivates structural regularity and texture legibility, including text and logos.
• Digital capture of cluttered real environments such as tabletops and indoor scenes, where objects are partly occluded and never isolated.

Industry relevance. The work comes from NVIDIA and Simon Fraser University, targets practical capture conditions rather than lab setups, and reports time budgets — the 0.5-minute unrefined result already beating a 20-minute from-scratch agent run is a direct cost argument. The agent stage, however, depends on a commercial multimodal model (GPT-6-Astra) accessed via a command-line agent, which the paper itself frames as time-consuming and costly when used from scratch.

Future Directions

• Reducing dependence on upstream estimates. Masks, cameras, and depth are often estimated (SAM3 masks on ScanNet++, Pi3X cameras and depth), and the ablation attributes the mixer's gain to robustness against such errors. Further work could tighten this coupling.

• Metric-scale reconstruction. The paper states that recovering metric dimensions requires metrically scaled input depth, leaving scale accuracy dependent on the depth source rather than on the model.

• Cheaper or lighter agentic refinement. Refinement uses GPT-6-Astra with extra-high reasoning effort and a ten-minute budget per object; the paper notes that from-scratch agent construction is slow and expensive with commercial APIs, so more efficient refinement is an open direction.

• Better reflecting downstream value in metrics. The authors observe that agentic edits are local and only slightly worsen CD, F1, and IoU on HANDAL, while benefiting simulation in ways the quantitative metrics cannot fully capture — pointing to a need for evaluation that measures simulation readiness.

Target Audience

Researchers and engineers working on 3D generative models, single- and multi-view object reconstruction, and scene-level asset creation; practitioners building simulation or content pipelines who need posed, textured objects from ordinary photos; and readers interested in how agentic editing loops can be grounded on a generative model's output rather than replacing it. The paper is dense with model names and metrics, so readers without a 3D vision background will find the methodology section demanding.

Authors’ abstract

Reconstructing complete, scene-aligned 3D objects from casual images requires integrating sparse, uncertain observations and inferring surfaces hidden by occlusions. We present GATOR, a generative and agentic framework that recovers textured object assets and their scene-relative pose from one or more images. Our local modality mixer couples patch-aligned RGB, target-mask, and pointmap features before cross-view reasoning, preserving scene context while distinguishing the target from its surroundings. Text-guided semantic conditioning complements these spatial cues with category names and object captions through stage-specific adapters for structure, geometry, and appearance generation. The generated asset initializes a multimodal agent, providing instance-specific geometry and pose for targeted structural and texture refinement through an observation-guided edit-render-review loop. Across synthetic objects, cluttered tabletops, and indoor scenes, GATOR achieves strong geometric and appearance fidelity while recovering scene-relative pose from sparse observations. Time-budget comparisons and scene-level simulation further demonstrate the reconstruction efficiency and simulation readiness. Project page: https://research.nvidia.com/labs/lpr/gator/