GENA3D: Generative Amodal 3D Modeling by Bridging 2D Priors and 3D Coherence

AI summary

Overview

Research area: Computer vision — amodal (occlusion-complete) 3D object generation from sparse, unposed views.

Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching generative models, cross-attention conditioning, multi-view stereo (MVS), and structured-latent 3D representations.

Scope: The paper proposes a conditional 3D generation framework that combines 2D generative completion priors with explicit 3D geometric conditioning to produce complete, occlusion-free 3D objects from sparse and partially occluded inputs.

Authors: Junwei Zhou and Yu-Wing Tai (Dartmouth College). arXiv:2511.21945v3 [cs.CV], CC BY 4.0. Project page: https://colezwhy.github.io/gena3d/

What This Paper Is About

In real-world captures, objects are frequently seen from arbitrary viewpoints with parts hidden behind other objects or structures, so any system that wants a full 3D object must infer geometry and appearance that was never directly observed. Existing methods split into two unsatisfying camps: approaches that work directly in 3D stay geometrically consistent but are not very expressive at inventing missing content, while 2D amodal completion produces convincing appearance but does not guarantee that the completed views agree in 3D. GENA3D's goal is to get both at once — plausible generative imagination plus multi-view and spatial coherence — from as few as one to four sparse, unposed views.

Key Contributions

1. GENA3D framework. A generative amodal 3D modeling framework that recovers complete geometry and appearance from sparse, occluded inputs by bridging 2D priors and 3D coherence during generation.
2. View-Wise Cross Attention (VWCA). A parallelized multi-view conditioning mechanism that performs cross-attention against each view separately, then fuses the resulting latents with a visibility-weighted average, explicitly countering "view dominance" (later views overwriting earlier structure) and "geometric drift."
3. Stereo-Conditioned Cross Attention (SCCA). A geometry-aware attention module that injects partial stereo (MVS) point-cloud structure directly into the attention logits through a Geometry-Guided Gating MLP, so geometric cues act as attention regulators rather than passive extra tokens.
4. Comprehensive validation. Experiments on synthetic (GSO) and real-world (COCO, Mip-NeRF 360, Hypersim) data show superior generation fidelity, completeness, and consistency versus sparse-view and amodal 3D generation baselines, plus ablations of each module, view count, 2D completion method, and MVS error robustness.

Main Findings

• Best overall amodal 3D generation on GSO. At 4 views, GENA3D reaches FID 30.73, KID 0.43, CLIP 82.53, MMD 5.48‰, COV 39.48%, and PCS 67.0, compared with Amodal3R (35.15 FID, 0.43 KID, 82.04 CLIP, 5.51 MMD, 38.83 COV, 46.2 PCS) and TRELLIS with OAAC (45.41 FID, 2.79 KID, 78.17 CLIP, 5.94 MMD, 34.42 COV, 23.2 PCS). GENA3D is also best at 1 view (33.91 FID, 82.23 CLIP, 39.19 COV, 61.9 PCS) and 2 views (32.12 FID, 82.33 CLIP, 39.08 COV, 66.3 PCS).
• Strong gains on the perceptual VLM metric. PCS (0–100 rating by Qwen3-VL 32B) jumps from 46.3 (Amodal3R) and 46.5 (SAM3D) to 61.9 for GENA3D at 1 view, and from 46.2 to 67.0 at 4 views.
• Better preservation of visible content. Visibility-masked faithfulness improves with more views: GENA3D reaches SSIM 0.832 / PSNR 16.52 / LPIPS 0.231 at 1 view, 0.837 / 16.99 / 0.212 at 2 views, and 0.838 / 16.98 / 0.208 at 4 views, versus Amodal3R at 0.814 / 16.41 / 0.241 (1 view) and TRELLIS at 0.674 / 13.66 / 0.357 (4 views).
• Both proposed attention modules matter. Removing Stereo-Conditioned Cross Attention costs the most (FID +10.80, MMD +0.41, COV −3.82); removing View-Wise Cross Attention costs +6.42 FID and −0.74 COV; removing the Geometry-Guided Gating MLP costs +0.55 FID and −1.40 COV; removing all proposed modules costs +14.11 FID, +0.44 MMD, and −4.66 COV relative to the full model.
• Performance scales with view count up to semi-dense settings. FID falls from 33.91 (1 view) to 32.12 (2), 30.73 (4), 29.56 (10), and 29.43 (20 views), with CLIP rising from 82.23 to 82.95 at 20 views — indicating stability well beyond the sparse regime.
• The framework is agnostic to the 2D completion front-end. With two views, OAAC (FID 32.12, COV 39.08), Flux inpainting (33.07, 39.49), and pix2gestalt (38.61, 37.26) all yield comparable, consistent results; stronger completers give better numbers.
• 2D completion alone is not enough. Equipping Amodal3R with the same OAAC front-end degrades it (FID 35.34 → 36.85; MMD 5.52 → 5.76; COV 38.58 → 37.46), while GENA3D with matched 2D completion outperforms Amodal3R by 4.81 FID and 1.70 COV (GENA3D: 32.04 FID, 5.50 MMD, 39.16 COV).
• Robust to severe occlusion. On 500 randomly selected GSO objects, GENA3D degrades gracefully from 20% to 70% occlusion: FID 32.28 → 37.22 with 1 view and 31.98 → 37.16 with 2 views; CLIP stays in the 81.44–82.54 range.
• Tolerant of MVS errors. When reconstructed stereo geometry is implausible or misaligned, the model can rely less on stereo conditioning and self-correct using its generative 3D prior (Fig. 7).
• Modest inference cost. GENA3D typically generates a single 3D object within 15 seconds, a slight overhead versus TRELLIS and Amodal3R that the authors consider worthwhile.

Methodology in Plain English

The pipeline is a conditional 3D generation process built from a few deliberately staged parts:

1. Get 2D amodal completions. From K sparse, unposed views (typically 1–4) of an object, the method uses vision foundation models (SAM, Florence 2) to obtain visibility masks and a text prompt, then applies an existing occlusion-analysis and 2D amodal completion strategy (denoted OAAC) to each view. This produces inconsistent but appearance-rich completed images of the object, along with visibility masks and occlusion masks.
2. Fuse the views without letting one dominate. Rather than conditioning sequentially (where later views overwrite earlier hypotheses) or naively concatenating features, VWCA runs cross-attention between the current 3D latent and every view's DINOv2 feature in parallel, then averages the resulting latents with weights proportional to each view's visibility ratio (visible area divided by visible-plus-occluded area), followed by a LayerNorm. The base generator's tendency to produce objects in canonical poses is exploited as a prior to make this pose-invariant fusion sensible.
3. Anchor the generation in real geometry. An MVS model (π³) turns the sparse views into point clouds; points falling inside the visibility masks are collected into a partial object point cloud, normalized, voxelized at grid size 1/64, encoded by a Sparse Structure VAE, and turned into geometry tokens. A Geometry-Guided Gating MLP converts those tokens into a gate that is added to the attention logits, so the geometry actively modulates attention instead of merely being concatenated as extra tokens.
4. Complete the structure and decode. The resulting sparse structure passes through a pretrained amodal structure-latent Transformer (from Amodal3R) for detailed structured latent generation, and is then decoded into a complete, occlusion-free 3D object.
5. Train with synthetic occlusion. Meshes from object-centric 3D datasets are converted to watertight meshes and given 3D-consistent occlusion masks by randomly growing from a seed face outward to a randomly sampled coverage ratio (20%–60%). Rendered views, masks, and controlled mask distortions produce the training quadruples. The trainable sparse-structure stage (stage 1) is optimized with a standard conditional flow matching objective, sampling 1–4 views per batch and generating stereo point clouds online.

Why This Matters

Impact on research. The paper reframes amodal 3D generation as a problem of coupling two sources of information — expressive 2D generative priors and explicit 3D geometric structure — rather than choosing between them. It shows that geometry can be used as an attention regulator inside a generative model instead of as a passive conditioning signal, and demonstrates that a 3D generator trained only on object-level occlusion data can generalize to real-world, unposed, scene-level imagery.

Real-world applications (as identified in the paper):

• Robotic interaction, where a robot needs the full shape of a partially occluded object to grasp or manipulate it.
• AR/VR content creation, producing complete 3D assets from casual sparse captures.
• Embodied AI perception, where agents must reason about objects they never fully see.
• Object-level 3D scene understanding and scene synthesis from sparse-view, heavily occluded inputs.

Industry relevance. The method integrates cleanly with off-the-shelf components (2D amodal completion models, π³-style MVS, pretrained structure-latent Transformers), which lowers the barrier to adoption in production pipelines for asset creation, digital twins, and simulations. Its roughly 15-second per-object inference and scene-level proxy — reconstructing the whole scene's stereo once and extracting per-object point clouds via visibility masks — matter for practical throughput.

Future Directions

• Close the plausibility-versus-truth gap. The authors acknowledge the model infers missing regions from learned priors, so outputs "may look plausible but not always match the ground-truth geometry." Ground-truth-consistent amodal hallucination remains open.
• Scale beyond object-level occlusion training. Training uses only object-level occlusion data with simulated 2D amodal completion; how far this transfers to full scene-level amodal generation is not fully established.
• Exploit stronger 2D front-ends. Results with Flux inpainting and pix2gestalt suggest the framework improves as 2D completion improves, motivating tighter integration with more advanced amodal reasoning models.
• Handle harder MVS failures and efficiency. The reported robustness to misaligned or overlapping MVS geometry is qualitative; a systematic error-propagation study is presented but the provided content truncates that table, and the added inference cost relative to TRELLIS and Amodal3R leaves room for optimization.

Target Audience

Researchers and practitioners in 3D generative modeling, amodal perception, and sparse-view reconstruction; engineers building asset-generation or scene-understanding pipelines for robotics, AR/VR, and embodied AI; and graduate students who already understand diffusion/flow-matching generation and multi-view geometry and want a concrete example of how to combine 2D generative priors with explicit 3D structural conditioning.

Authors’ abstract

Generating complete 3D objects under partial occlusions (i.e., amodal scenarios) is a practically important yet challenging problem, as large portions of object geometry are unobserved in real-world scenarios. Existing approaches either operate directly in 3D, which ensures geometric consistency but often lacks generative expressiveness, or rely on 2D amodal completion, which provides strong appearance priors but does not guarantee reliable 3D structure. This raises a key question: how can we achieve both generative plausibility and geometric coherence in amodal 3D modeling? To answer this question, we introduce GENA3D (GENarative Amodal 3D), a framework that integrates learned 2D generative priors with explicit 3D geometric reasoning within a conditional 3D generation paradigm. The 2D priors enable the model to plausibly infer diverse occluded content, while the 3D representation enforces multi-view consistency and spatial validity. Our design incorporates a novel View-Wise Cross-Attention for multi-view alignment and a Stereo-Conditioned Cross-Attention to anchor generative predictions in 3D relationships. By combining generative imagination with structural constraints, GENA3D generates complete and coherent 3D objects from limited observations without sacrificing geometric fidelity. Experiments demonstrate that our method outperforms existing approaches in both synthetic and real-world amodal scenarios, highlighting the effectiveness of bridging 2D priors and 3D coherence in generating plausible and geometrically consistent 3D structures in complex environments.