A Unified Framework for Image-to-3D Part Generation via Variable Granularity

AI summary

Overview

Research area: Computer Vision / 3D generative modeling — specifically image-to-3D part generation, where a single image plus user-supplied geometric hints is turned into a set of separable 3D parts.

Technical level: Advanced. The paper builds on Diffusion Transformers, rectified flow, DINOv2 features, and differentiable (partial) optimal transport; familiarity with diffusion-based 3D generation is assumed.

Scope: The paper introduces FlexPart, one model that accepts point, box, or mask prompts of varying granularity for controllable part-level 3D generation, and reports part-level and whole-mesh benchmarks on PartVerseXL and PartObjaverse-Tiny.

What This Paper Is About

Existing image-to-3D part generators are built for a single kind of user input: some need dense pixel masks or explicit box layouts, others only take a part count and give up spatial control. This forces users into one workflow and ignores the fact that coarse and fine prompts carry complementary information. The paper asks whether a single model can accept sparse points, 2D boxes, and 2D masks interchangeably — and whether training on all of them jointly makes each mode better than training on it alone.

Key Contributions

1. FlexPart, a unified multi-granularity framework. The authors integrate point, box, and mask prompts into a single Diffusion Transformer (DiT) through Gated Adaptive Layer Normalization, so one trained model supports all three input types without retraining. They describe it as the first framework to unify point, box, and mask prompts in a single DiT for controllable 3D part generation.

2. Asymmetric Geometric Guidance. A training-only mechanism that uses differentiable partial optimal transport to let strong priors (masks and boxes) supervise the ambiguous representations produced by sparse point prompts, transferring structural knowledge from dense to sparse conditions.

3. Demonstrated cross-modal synergy. Joint training across granularities beats single-modality training baselines under every evaluation condition tested, and stronger prompts improve weaker ones rather than conflicting with them.

4. State-of-the-art part-level results with competitive whole-mesh quality. Under mask conditioning on PartVerseXL, FlexPart reduces Part CD by 16.6% relative to TripoSG + SAM3D and by 10.5% relative to OmniPart.

Main Findings

• Best part-level geometry under masks: On PartVerseXL, FlexPart with mask input reaches Part CD 0.2938, Part IoU_V 0.2049, and Part IoU_B 0.2840, versus OmniPart at 0.3283 / 0.1961 / 0.2831 and TripoSG + SAM3D at 0.3523 / 0.1727 / 0.1896. On PartObjaverse-Tiny the corresponding FlexPart numbers are 0.3560 / 0.1571 / 0.2778 against OmniPart's 0.3681 / 0.1568 / 0.2649.

• Graceful degradation as prompts get weaker: Relaxing from masks to boxes changes Part CD by only 3.5% on PartVerseXL (0.2938 to 0.3042). Even with points only, FlexPart reaches Part CD 0.3334, better than the structurally unconstrained PartPacker (0.3933) and PartCrafter (0.5235).

• Competitive assembled meshes: On PartVerseXL, FlexPart achieves Mean CD 0.1418, F-Score 0.7877, and ULIP 0.2206, which the authors state surpasses OmniPart's 0.1547 by 8%. On PartObjaverse-Tiny, FlexPart scores Mean CD 0.1587, F-Score 0.7510, ULIP 0.2350 — close to but not better than several baselines, e.g. TripoSG at 0.1504 / 0.7811 / 0.2410.

• Joint training beats single-modality training: Training on all three prompt types improves point-only inference by 7.9% in IoU_V and 6.1% in CD relative to a point-only training baseline. Under mask evaluation, Part CD drops from 0.3224 (mask-only training) to 0.2938 (all three modalities).

• Asymmetric Geometric Guidance helps sparse inputs most: Adding the alignment loss under point prompts improves Part IoU_B from 0.2323 to 0.2506 (a 7.9% gain) and Part IoU_V from 0.1811 to 0.1902 (5.0%). Under masks, Part CD improves from 0.3011 to 0.2938 with no reported conflict.

• Robust to noisy and mixed prompts: Perturbing 2D box position and size by up to 20% moves Part CD from 0.3042 to 0.3171 and Part IoU_V from 0.2078 to 0.2032. Supplying boxes for half the parts and points for the rest gives Part CD 0.3287; mixing masks for half the parts with boxes for the other half gives 0.3190, even though that combination was unseen in training.

• Point placement matters less with stronger prompts: With points only, moving from a Center to an Edge placement raises Part CD from 0.3302 to 0.4190 and lowers Part IoU_V from 0.1905 to 0.1342. Adding box or mask prompts confines Part CD to 0.3036–0.3178 and 0.2995–0.3087 across the four placement strategies.

• Handles explicit 3D boxes too: On PartObjaverse-Tiny with 3D box conditioning, FlexPart beats FullPart on whole-mesh metrics (CD 0.1385 vs 0.1625, F-Score 0.8158 vs 0.7577, ULIP 0.2209 vs 0.1775) and on Part CD (0.2356 vs 0.2488) and Part IoU_V (0.2653 vs 0.2064), while FullPart leads on Part IoU_B (0.4370 vs 0.4027).

Methodology in Plain English

FlexPart starts from a pretrained TripoSG generative backbone (a Diffusion Transformer over 3D shape latents) and fine-tunes it in two stages. Each part is described by a conditioning vector built from whichever prompts the user provides.

Encoding the prompts. Points are encoded with Gaussian Fourier features plus a small MLP; boxes encode their top-left and bottom-right corners separately with a learnable corner-type embedding inspired by SAM; masks go through a small CNN with global average pooling. During training, any prompt type can be randomly replaced by a learnable "null" embedding with a set dropout probability, so the model learns to work with whatever subset of prompts is available. The three embeddings are concatenated and passed through a fusion MLP into one conditioning vector per part.

Injecting the condition. That vector feeds a Gated AdaLN module that predicts scale, shift, and gate values applied to each transformer layer's normalized hidden states. Image features from DINOv2 enter separately through cross-attention, so cross-attention supplies global image context while the gated modulation governs part-level geometry. A part-aware attention scheme treats each part as a separate sample in the batch.

Transferring knowledge from dense to sparse prompts. In the Asymmetric Geometric Guidance branch, point embeddings (the "student") and box/mask embeddings (the "teacher") are both projected into a shared visual space via a Visual Bridge with cross-attention over image features. A stop-gradient is applied to the teacher so the alignment loss does not update it. A similarity matrix combining box and mask similarities (weighted by α) is augmented with a learnable "dustbin" score and solved with log-domain Sinkhorn to obtain a partial optimal transport plan. The training loss combines a conditional rectified-flow velocity-matching term with an auxiliary term that maximizes the diagonal transport probabilities. The branch is used only during training, adding no inference cost.

Training data and setup. Training uses a composite set from Objaverse and PartVerseXL (about 250K 3D models, of which roughly 180K are effectively used due to batch-wise part packing). Stage 1 covers objects with 1–8 parts at 512 tokens per part, with 0.5 dropout on boxes and masks. Stage 2 covers 2–16 parts at 1024 tokens per part, with dropout 0.1 for points and 0.7 for masks, and a mixed box-dropping strategy (0.25 full dropout, 0.25 masking half the parts). Training runs on 8 NVIDIA H20 GPUs with batch size 256 and learning rates of 1×10⁻⁴ and 5×10⁻⁵ for the two stages.

Evaluation. Held-out test sets are drawn from PartVerseXL (100 shapes with at most 16 annotated parts) and PartObjaverse-Tiny (all shapes with up to 16 parts). Part-level metrics are computed only over CD-based Hungarian-matched pairs, after filtering fully occluded ground-truth parts. Voxel IoU uses a distance-field occupancy definition at pitch h_vox = 0.03 to handle non-watertight meshes.

Why This Matters

For research: The paper reframes prompt granularity as a training signal rather than a fixed design choice, and shows that dense annotations can be used to regularize sparse-prompt learning through optimal transport. That is a reusable idea for any conditional generative model where annotation density varies.

Real-world applications:

• Animation and rigging: separable, semantically coherent parts are what a rigging pipeline needs; the paper motivates parts as "structurally detachable" and "spatially assemblable."
• Physical simulation: simulation requires per-component geometry rather than a single shell.
• Interactive 3D editing: users can select a component with a click (a point) or a rough box and regenerate just that piece.
• Digital asset creation: fast part-level authoring from a single product or object photo.

Industry relevance: A single deployed model serving point, box, and mask workflows lowers the barrier for non-expert users, who can start with a click and escalate to precise masks only when needed. Training-time-only guidance means no additional inference cost, which matters for production latency budgets.

Future Directions

• Occlusion reasoning: the authors flag occlusion handling as an open weakness — single-view 2D conditions give limited depth cues, and the model may place occluded parts incorrectly in visible regions.
• Beyond single-view and 2D prompts: the 3D-box variant shows the gated modulation extends to explicit 3D constraints; extending further to multi-view or full 3D layout control is a natural next step.
• Replacing cascaded SAM pipelines: the authors argue end-to-end heterogeneous conditioning avoids the accumulated errors of converting sparse prompts to masks with a 2D segmenter; quantifying that gap systematically is left largely to the appendix.
• Closing the whole-mesh gap: on PartObjaverse-Tiny, native part-based models do not yet match the best holistic generators on Mean CD, F-Score, and ULIP, so reconciling part control with full-object fidelity remains open.

Target Audience

Researchers and graduate students working on 3D generative models, diffusion/flow-matching architectures, and controllable content creation. It is also relevant to graphics and AR/VR engineers who need editable, part-structured assets in a production pipeline, and to practitioners interested in multi-modal conditioning and knowledge transfer between annotation granularities. Readers without a background in diffusion models or optimal transport will find the methodology section demanding, though the experimental tables are readable on their own.

Authors’ abstract

Controllable 3D part generation is pivotal for digital asset creation, yet existing methods lack the flexibility to accommodate diverse user workflows requiring varying levels of control. In this paper, we introduce FlexPart, a unified image-to-3D part generation framework that supports variable-granularity geometric conditions within a single model. By seamlessly integrating heterogeneous inputs, ranging from sparse points to dense bounding boxes and masks, via gated adaptive modulation, FlexPart unlocks a powerful cross-modal synergy. We also propose an asymmetric geometric guidance mechanism that leverages strong spatial priors to align representations across granularities, improving geometric consistency and facilitating inter-task synergy. Under mask conditioning on PartVerseXL, FlexPart reduces Part CD by 16.6% relative to TripoSG + SAM3D and by 10.5% relative to OmniPart, while maintaining competitive whole-mesh quality. The code can be found at https://github.com/Gucci233/FlexPart