SCULPT: Subtractive Composition for 3D Part Generation

AI summary

Overview

Research area: Computer Vision / 3D generative modeling — specifically part-aware 3D object generation, the task of producing a complete 3D asset that is also explicitly decomposed into editable semantic components.

Technical level: Advanced. The paper assumes familiarity with latent diffusion / rectified-flow generative models, sparse voxel (O-Voxel) representations, ControlNet-style conditioning branches, and 3D part segmentation benchmarks.

Scope (one sentence): SCULPT reformulates part-aware 3D generation as a repeated "subtractive" operation — at each step a trained joint split predictor denoises one extracted part together with the remaining object — and is evaluated on geometry at the part, semantic-group, and assembled-object levels on PartObjaverse.

What This Paper Is About

Existing part-aware 3D generators either segment an object after it has been fully generated (so part boundaries are decided too late and new contact surfaces are never generated), or they synthesize parts additively from fixed slots, boxes, or token sequences and then try to reconcile them into a coherent whole (which often leaves gaps, interpenetrations, and material discontinuities at shared boundaries). SCULPT's goal is to keep the generative model inside the decomposition loop: given an image, it starts from a complete object latent and repeatedly carves out one part while explicitly carrying the updated remainder forward, so the boundary between part and remainder is decided during generation rather than imposed afterward.

Key Contributions

1. A subtractive composition formulation. Variable part cardinality is expressed through the length of a recurrent rollout of fixed-signature part–remainder splits, instead of a single variable-length prediction, a fixed number of output slots, or a pre-specified layout.
2. A joint split predictor built from decomposition flow transformer blocks. It adapts a pretrained holistic image-to-3D generator (TRELLIS.2) using image-conditioned joint denoising plus a remainder-conditioned control branch, so each extracted part and its remainder are generated in a shared trajectory.
3. Native-support constraints for subtractive generation. A sparse-support composition loss, inference-time support clipping, and empty-remainder termination keep the recurrent state inside the object being decomposed and preserve coverage of its current support. Part and remainder supports are allowed to overlap in an "interface shell" rather than being forced into a disjoint voxel partition.
4. An evaluation across three geometric levels. The paper reports state-of-the-art Chamfer distance (CD) at the part, semantic-group, and object levels on PartObjaverse, plus qualitative decompositions for four dataset images, one text-to-image-generated input, and one real-world photograph.

Main Findings

• Best CD and best F1@.05 at every evaluation level. SCULPT reports part-level CD 0.0107, F1@.1 0.8858, F1@.05 0.7599; semantic-group-level CD 0.0107, F1@.1 0.8851, F1@.05 0.7614; and object-level CD 0.0020, F1@.1 0.9839, F1@.05 0.9212 on PartObjaverse. It also reports the best object-level F1@.1.
• Improvement over the strongest direct part generator (OmniPart). Part-level CD drops from 0.0136 to 0.0107 and F1@.05 rises from 0.7025 to 0.7599. The gain survives assembly: object-level CD falls from 0.0032 to 0.0020 and F1@.05 rises from 0.8732 to 0.9212. The paper argues this shows that starting from a complete-object latent does not merely preserve the whole — it yields more accurate individual components than direct part generation.
• Improvement over the most competitive post-hoc decomposition baseline (TRELLIS.2+PartField). SCULPT reduces CD by 7.0% and 8.5% at the first two levels reported in that sentence; the third figure in the sentence is cut off in the supplied text. The underlying table values move from part-level CD 0.0115 to 0.0107 and object-level CD 0.0021 to 0.0020.
• One metric where it does not lead. At F1@.1, TRELLIS.2+PartField scores 0.8897 at the part level and 0.8903 at the semantic-group level, slightly above SCULPT's 0.8858 and 0.8851.
• Post-hoc baselines can degrade object fidelity. TRELLIS.2+PartField+HoloPart has worse part-level CD (0.0453) than TRELLIS.2+PartField (0.0115) and much worse object-level CD (0.0332 vs 0.0021). Object-level scores for TRELLIS+SAM3D and TRELLIS.2+SAM3D are marked unavailable in the table.
• Training scale and data. Supervision comes from PartVerse-XL, curated from Objaverse-XL. After filtering out assets with excessive component counts, extremely small components, missing PBR materials, or low-quality geometry and texture, the training set contains 37,425 objects and 330,455 supervised part–remainder splits, with training assets removed when their SHA-256 identifier matches an evaluation mesh.
• Termination is learned and bounded. The rollout ends when the remainder's sparse support becomes empty or after a fixed safety cap of K_max = 24 splits; if the cap is reached with a nonempty remainder, that remainder is decoded as one additional output rather than discarded.
• Recursive decomposition scales beyond the cap. In Figure 1(c), applying the split operation to previously extracted parts decomposes a complex asset into more than 100 fine-grained components.
• Reported generalization beyond the benchmark. Qualitative results are shown for four dataset images, one text-to-image-generated input, and one casually captured real-world photograph.

Methodology in Plain English

SCULPT begins by using a pretrained image-conditioned holistic generator (TRELLIS.2) to turn the conditioning image into a complete-object latent in a structured 3D latent space with three stages: sparse structure, geometry, and material. Everything — complete objects, parts, and remainders — lives in the same [-1, 1]^3 object frame, so no part has to be rescaled or registered before the pieces are combined.

The training data are part-annotated assets with a complete mesh and its part meshes. The parts are sorted lexicographically by their centroids along the z, x, y axes, giving one deterministic extraction order per asset. A complete mesh with K ordered parts yields K training examples, each of the form (conditioning image, current remainder; extracted part, updated remainder), plus a final example whose target remainder is empty — that last example is what teaches the model to stop. Crucially, the part and the remainder are voxelized independently, so their supports may overlap at the contact region instead of being forced into a disjoint voxel assignment.

The core module is a joint split predictor implemented as three stage-wise decomposition flow transformers, one per stage. At each split, the extracted part and the remainder are packed together, denoised as a single target on the union of their sparse supports, and then routed back to their respective supports. A parallel ControlNet-style control branch reads the current remainder and injects zero-initialized residuals into the joint denoising blocks; the joint blocks are initialized from TRELLIS.2 checkpoints, so training starts from a strong image-to-3D prior and adds the 3D remainder as a block-wise condition.

Two losses are used. A conditional flow-matching loss supervises both packed targets; a composition loss at the sparse-structure stage encourages the sigmoid probabilities of the predicted part and remainder to cover the current input support when combined as a differentiable union. Because that loss constrains the union and not the intersection, voxels may be active in both outputs — the overlapping boundary voxels the authors call an interface shell.

At inference, the rollout is a recurrence: threshold the two predicted occupancy maps with a fixed occupancy threshold (the numeric value is not reported), clip both supports to the previous remainder's support, mask the latents to those clipped supports, run the geometry and material stages, store the part, and pass the updated remainder to the next split. Each output latent is decoded independently, and the decoded meshes are united in the shared object frame with no per-part rescaling, snapping, or registration.

Why This Matters

Impact on research. SCULPT reframes part-aware 3D generation as a generative decomposition problem rather than a recognition problem over a frozen asset, and it does so without predicting the entire variable-cardinality part set in one shot. It also identifies a specific mechanism — shared denoising of part and remainder on the union of their native sparse supports — that a segment-then-reconstruct or generate-then-reconcile pipeline structurally cannot reproduce. The paper reports CD and F1 at three separate levels (part, semantic group, object) so that improvements in components cannot be hidden behind degradation of the assembled whole, a comparison protocol other part-generation work can reuse.

Real-world applications:

• Character rigging and animation. Parts extracted with separate geometry and material, all aligned in the original object frame, map directly onto limbs and other rigged components.
• Multi-material fabrication. Distinct material assignment to individual components is a native requirement of multi-material 3D printing and fabrication pipelines.
• Game, VFX, and interactive content authoring. Sub-parts can be edited, swapped, or reused independently while the assembled asset is expected to remain visually coherent.
• Digitizing physical objects. The paper shows a single casually captured photograph yielding a combined object, semantic segmentations, and separated textured 3D parts, which matters for AR/VR and e-commerce catalogues built from real photos.

Industry relevance. The method is built by adapting publicly released TRELLIS.2 checkpoints rather than training a generator from scratch, which lowers the barrier to adoption inside existing graphics pipelines. Its value proposition is directly commercial: assets that are coherent as wholes but structured for editing, material assignment, animation, and reuse. Note that this is an arXiv preprint (arXiv:2608.13541v1, cs.CV, 13 Aug 2026) with a project page; inference cost, wall-clock time, and any user study are not reported in the supplied text.

Future Directions

• Semantic versus geometric part ordering. Training uses a deterministic lexicographic ordering of part centroids along z, x, y. Whether a semantic or learned ordering changes the decomposition quality, the number of parts produced, or termination behavior is an open question the paper does not resolve.
• Scaling past the cap. The rollout is bounded at K_max = 24, and the paper reaches more than 100 components only by applying the split operation recursively to already-extracted parts. How far this recursion can be pushed, and what constraints break first, is not characterized.
• Rigorous treatment of the interface shell. Overlapping boundary voxels are proposed to avoid gaps and interpenetration, but the paper's geometry metrics do not isolate how much overlap occurs, whether the assembled union double-counts material, or how the shell behaves under animation.
• Beyond geometry-only benchmarking. Evaluation is on the 200-mesh PartObjaverse benchmark using CD and F1 at three thresholds; texture, material continuity at boundaries, and behavior across a broader range of object categories are argued through qualitative figures rather than quantitative metrics.

Target Audience

Researchers and practitioners in 3D generative modeling and graphics who are working on part-aware asset generation, structured 3D representations, or conditional control of pretrained generative backbones. It is also relevant to technical artists and pipeline engineers interested in assets that can be rigged, textured, and fabricated part by part, and to readers comparing direct part synthesis against segment-then-generate and generate-then-reconcile approaches under a shared evaluation protocol.

Authors’ abstract

Part-aware 3D generation aims to create digital assets that are coherent as complete objects while exposing structural parts for editing, material assignment, animation, and reuse. Existing methods impose this structure outside the native generation loop: segmentation-based methods partition an already generated shape, while additive methods synthesize parts from predefined layouts, boxes, or tokens and then reconcile them into a whole. The former preserves the generated geometry but fixes the object before part boundaries are determined; the latter exposes part cardinality but often leaves shared boundaries vulnerable to gaps, interpenetrations, and material discontinuities. In this paper, we propose SCULPT, a framework that addresses these challenges through subtractive composition. Given a complete object represented in a structured 3D latent space, SCULPT iteratively applies a joint split predictor to generate one extracted part together with the remaining object. The predictor performs a coupled denoising process conditioned on both the image and the current 3D state, so the extracted part and updated remainder are generated together rather than reconciled after generation. The joint split predictor processes both outputs on the union of their native sparse 3D supports, allowing neighboring supports to overlap rather than imposing a disjoint voxel partition. The rollout ends when the remainder support becomes empty or reaches a fixed safety cap, allowing the number of generated parts to adapt to each object within that bound. Extensive experiments demonstrate state-of-the-art geometry on PartObjaverse while preserving strong complete-object reconstruction after part assembly. Results on four dataset images, one text-to-image-generated input, and one real-world photograph further show fine-grained textured part decomposition beyond the benchmark.