RelaxFlow: Text-Driven Amodal 3D Generation

AI summary

Overview

• Research area: Computer vision and 3D generative modeling, combining image-to-3D generation, amodal (occlusion-aware) completion, and text-guided controllable generation.
• Technical level: Advanced. The paper combines a training-free inference framework with formal analysis of generative ODEs, frequency-domain arguments, and Wasserstein-distance bounds.
• Scope: The paper formalizes text-driven amodal 3D generation — using a text prompt to steer how occluded parts of an object are completed while strictly preserving the visible evidence — and proposes RelaxFlow, a training-free dual-branch method, plus two diagnostic benchmarks.

What This Paper Is About

When an object is heavily occluded, a single visible view often does not contain enough information to determine what the object even is, so a feedforward image-to-3D model must guess. Existing models tend to collapse to a single observation-overfitted shape, while optimization-based methods that follow text prompts tend to distort or over-smooth the visible evidence. This paper formalizes text-driven amodal 3D generation, where a user's text prompt resolves the ambiguity in unseen regions while the visible pixels remain strictly preserved, and proposes a method that satisfies both constraints at once.

Key Contributions

1. Task formalization. The paper defines text-driven amodal 3D generation, a setting where the same observation admits multiple plausible completions and text prompts must resolve the occlusion-induced ambiguity without sacrificing observation fidelity.
2. RelaxFlow framework. A training-free dual-branch inference framework that decouples control granularity: an Observation Branch with rigid control for visible pixels, and a Semantic-Prior Branch with relaxed structural control, combined through a Multi-Prior Consensus Module and a visibility-aware fusion strategy.
3. Theoretical proof of low-pass relaxation. The authors prove that the relaxation applied to the prior branch is equivalent to applying a low-pass filter on the generative vector field, and derive a spectral analysis showing it strictly reduces semantic estimation error, plus a Wasserstein-2 bound on the distance to the ground-truth distribution.
4. Two diagnostic benchmarks. ExtremeOcc-3D and AmbiSem-3D are introduced to evaluate whether text can disambiguate unseen structure without compromising adherence to observed evidence.

Main Findings

• ExtremeOcc-3D gains over both backbones. On TRELLIS, Point-FID drops from 141.48 to 97.79, CLIP_txt rises from 23.14 to 24.09, FID falls from 122.68 to 100.75, and CLIP_img rises from 0.78 to 0.80. On SAM3D, CLIP_txt rises from 24.08 to 27.26, Point-FID falls from 100.38 to 81.11, FID falls from 50.73 to 39.44, CLIP_img rises from 0.84 to 0.87, and LPIPS improves from 0.54 to 0.51.
• Competitive preservation of visible evidence. The paper reports that LPIPS and CLIP_img on SAM3D remain competitive with the unmodified backbone, which the authors interpret as evidence that RelaxFlow preserves the observed input view rather than drifting from it.
• AmbiSem-3D results. RelaxFlow reaches CLIP_img 0.87 and CLIP_txt 27.23, compared with SAM3D at 0.85 / 26.29, TRELLIS (multi-view) at 0.80 / 26.59, SDXL + TRELLIS at 0.81 / 26.76, and SDXL + SAM3D at 0.79 / 26.71.
• User study. With 32 volunteers evaluating all 21 cases, RelaxFlow was preferred 68.52% overall, with 73.91% preference for Text–Image Alignment and 63.13% for 3D Fidelity, versus single-digit to low-double-digit percentages for all baselines.
• Low-pass relaxation is load-bearing. Removing it degrades Point-FID from 81.1 to 87.1 on ExtremeOcc-3D with the SAM3D backbone.
• The visibility mask matters more. Disabling the visibility mask degrades Point-FID from 81.1 to 92.3, the largest drop among the ablated components.
• Hyperparameter sensitivity. A cutoff of ρ = 0.4 gives Point-FID 86.5 and ρ = 1.0 gives 89.9, both worse than the default ρ = 0.2 configuration at 81.1. Stronger relaxation with σ = 2.5 gives 95.2, worse than the default σ = 1.0.
• Generated priors remain competitive. Using Z-Image-generated priors instead of retrieved ones gives Point-FID 82.7 versus 81.1 for the full method.
• Prior count has a sweet spot. Moderate N improves consensus-based disambiguation, but excessive priors introduce conflicting details, as shown in the paper's Figure 6b.
• Qualitative failure modes of baselines. The paper reports that TRELLIS drifts from the observation, SAM3D responds weakly to the textual prompt, and 2D-based editing pipelines introduce geometric artifacts, while multi-view TRELLIS drifts due to conflicting views.

Methodology in Plain English

The authors treat generation as solving an ordinary differential equation (ODE) where the velocity at each step is a blend of two forces: a hard observation constraint that anchors the output to the visible pixels, and a soft semantic prior that fills in what is missing. They formalize this as a weighted sum, (1 − α_t) times the observation velocity plus α_t times the prior velocity.

Their key insight is that these two forces need different levels of strictness. The observation needs rigid, rigid pixel-level control, while the prompt should only guide coarse structure and tolerate local variation. So they run two parallel branches on the same intermediate state and blend the results.

The Semantic-Prior Branch is where the interesting trick lives. Modern feedforward 3D generators expect image-derived visual tokens, not free-form language, so the authors convert the text prompt into a small set of reference images (N = 3 by default) — either retrieved from a data pool or generated with an off-the-shelf text-to-image model such as Z-Image. Because these references share the intended attribute but differ in incidental appearance, concatenating their tokens and feeding them through a single pass of cross-attention makes consistent attributes accumulate attention while idiosyncratic details get diluted. That is the Multi-Prior Consensus.

Then they blur the cross-attention logits in the prior branch with a separable Gaussian filter (implemented as two 1D convolutions along the query and key dimensions, with ±3σ support, controlled by σ = 1.0) before the softmax. Theoretically this is equivalent to low-pass filtering the generative vector field, which suppresses high-frequency instance-specific details and leaves a coarse semantic corridor that only constrains global geometry.

Fusion is time-dependent and spatially aware. A linear cutoff schedule lets the prior steer the early steps (where the global semantic mode is decided) and then yields entirely to the observation branch after a fraction ρ = 0.2 of the steps. In the second (SLAT) stage, per-voxel visibility is estimated by projecting voxel centers to the camera and comparing depth against a z-buffer, producing a soft visibility weight that keeps the prior from overpainting surfaces already supported by the observation.

The method is instantiated on two feedforward generators, TRELLIS and SAM3D, both of which use a two-stage pipeline: sparse structure (SS) for geometry on a 64³ grid, and structured latent (SLAT) for texture and detail. TRELLIS lacks pose estimation, so the visibility mask is disabled for that backbone.

Why This Matters

The paper reframes occlusion handling in image-to-3D generation from "learn the most likely completion" to "let the user specify the completion." Because RelaxFlow is training-free and attaches to existing backbones, it offers a route to controllable 3D generation without retraining generators on new data.

• Content creation and e-commerce: turning a single partially visible product photo into a complete 3D asset matching a described category.
• AR/VR and scene reconstruction: completing furniture or objects captured from occluded viewpoints in indoor scans without re-scanning.
• Robotics and manipulation: reasoning about the full physical extent of a partially hidden object when the intended category is known or specified.
• 3D asset pipelines in design: steering an ambiguous observation toward a specific intended object type rather than accepting a model's default guess.

Industry relevance: the work targets practitioners who already deploy feedforward image-to-3D models and want prompt-based control without finetuning per model. The reported memory overhead is modest and the low-pass relaxation is implemented with efficient 1D convolutions, though the extra branch evaluations do increase runtime.

Future Directions

• Closing the proxy gap. The Wasserstein bound includes a term δ_prior capturing the mismatch between visual prior tokens and the true intent. Multi-prior consensus reduces it empirically, but the paper does not report a way to eliminate it.
• Reducing runtime cost. The paper reports that the extra branch evaluations increase runtime; a cheaper alternative to running a second branch at every solver step is not reported.
• Adaptive prior selection. The paper shows moderate N helps and excessive N hurts, but no automatic method for choosing or filtering priors is reported.
• Extension beyond the evaluated backbones. RelaxFlow is instantiated on TRELLIS and SAM3D, and the comparison in Appendix E against commercial 2D editing, video generation, multi-view generation, and direct text+image 3D pipelines is cited but its full results are not included in the provided content.

Target Audience

Researchers and engineers working on 3D generative models, diffusion and flow-matching methods for 3D, amodal perception, and controllable content generation. Readers who want the theoretical framing of attention smoothing as low-pass filtering on a vector field will benefit most, as will practitioners seeking a training-free way to add text controllability to an existing feedforward image-to-3D pipeline. The paper assumes familiarity with ODE-based generative modeling, cross-attention, and Wasserstein distances.

Authors’ abstract

Image-to-3D generation faces inherent semantic ambiguity under occlusion, where partial observation alone is often insufficient to determine object category. In this work, we formalize text-driven amodal 3D generation, where text prompts steer the completion of unseen regions while strictly preserving input observation. Crucially, we identify that these objectives demand distinct control granularities: rigid control for the observation versus relaxed structural control for the prompt. To this end, we propose RelaxFlow, a training-free dual-branch framework that decouples control granularity via a Multi-Prior Consensus Module and a Relaxation Mechanism. Theoretically, we prove that our relaxation is equivalent to applying a low-pass filter on the generative vector field, which suppresses high-frequency instance details to isolate geometric structure that accommodates the observation. To facilitate evaluation, we introduce two diagnostic benchmarks, ExtremeOcc-3D and AmbiSem-3D. Extensive experiments demonstrate that RelaxFlow successfully steers the generation of unseen regions to match the prompt intent without compromising visual fidelity.