KaiNinja: Extending Native 3D Generators to the Part Level

AI summary

Overview

Research area: 3D generative modeling — specifically part-level image-to-3D generation built on top of native 3D generators (TRELLIS.2 and its O-Voxel representation).

Technical level: Advanced. The paper assumes familiarity with rectified flow / diffusion models, sparse voxel latents, dual contouring, classifier-free guidance, and the two-stage cascade architecture of TRELLIS.2.

Scope: The paper introduces KaiNinja, a part-level extension of TRELLIS.2 that packs an object's parts into two interleaved O-Voxel volumes so that parts can be generated and separated in one pass with no mask, no segmenter, and no per-object optimization.

What This Paper Is About

Native 3D generators such as TRELLIS.2 turn a single image into high-fidelity geometry with materials, but they return one fused mesh, while editing, rigging, and simulation all need each part as a separate mesh. Existing routes to parts either segment a finished mesh and regenerate its parts (slow, and bounded by segmentation accuracy) or build part structure into generation using 2D masks or predicted bounding boxes (which fix the layout before any 3D reasoning happens). The paper's goal is a fast, simple extension of the pretrained TRELLIS.2 that produces parts directly from one image, with the number of parts not fixed in advance.

Key Contributions

1. The first native part-level extension of TRELLIS.2, the state-of-the-art object-generation foundation model. The recipe starts from the pretrained weights and treats the two stages differently: per-stream weights connected by new cross-volume blocks for the layout flow, and an untouched backbone with volume-restricted attention scoping for the refinement flow.
2. A representation-level argument for why parts require changing the representation: one volume cannot hold a part interface, because the O-Voxel grid stores one sheet of surface per voxel, so two touching contact faces collapse into one at any resolution. The remedy is dual-volume packing on the generator's own sparse voxel grid, which keeps open surfaces, UVs, materials, and PBR attributes intact.
3. The first use of agent-authored 3D assets to train a 3D generative model. The released Articraft-10K collection is used, whose assets are built by programs that assemble each part from primitives, so part labels come from authoring rather than annotation and carry no labeling noise.
4. State-of-the-art part-level image-to-3D results on a held-out test set spanning four data sources, with no mask or segmenter anywhere in the pipeline.

Main Findings

• No segmenter in the pipeline. Parts are read off as connected components of the two decoded volumes. The number of parts is never predicted by a separate module; it follows from the layout the model generates, and there is no upper bound on the number of parts because any number fits into a fixed pair of streams.
• Speed. At 512 generation resolution, KaiNinja produces a part-separated asset in about 24 seconds per object on one H100.
• Accuracy against part-generation pipelines of other paradigms. It lowers whole-object Chamfer distance by 40% and raises strict part F-score by 16%. Against the strongest baseline, Hunyuan3D-2.1 + X-Part, it gains 0.10 F1^0.05_W and 0.09 F1^0.05_P at the strict threshold, with whole-object Chamfer distance 40% lower. The paper reports it is best on all seven metrics of Table 1 and on every entry of the per-source breakdown.
• The gap is widest at the strict threshold. The authors read this as evidence that the difference lies in fine geometry and boundary placement rather than in coarse layout.
• Whole-object fidelity improves over the same backbone fine-tuned on the same corpus. A 38% reduction in Chamfer distance is reported in the introduction. Giving TRELLIS.2@512 the same fine-tuning data raises it to 0.830 F1^0.05_W and cuts its failure rate to 1.8%; KaiNinja starts from the same pretrained weights and sees the same corpus yet reaches 0.919, another 0.089 above that, at about 2.5× the generation cost, while also delivering parts. It misses the shape outright on 0.7% of objects, against 3.9 to 6.8% for the zero-shot references.
• Part structure usually costs fidelity when reached from outside the generator. Four of the five part-producing baselines score at or below every zero-shot whole-object reference at the strict threshold (0.749 to 0.781 against 0.781 to 0.807). Only the Hunyuan3D-2.1 + X-Part cascade is higher, at 0.816, and only 0.009 above its own upstream generator.
• Packing may be a better representation of whole objects, not just a container for parts. The authors state they did not expect the whole-object gain. Two qualifications are offered: running TRELLIS.2 as a 1024 cascade does not help on this corpus (0.0364 against 0.0363), and TRELLIS and Hunyuan3D-2.1 share a mean but not a distribution — Hunyuan3D-2.1 has the better median CD_W, 0.0209 against 0.0235, and nearly twice the failure rate.
• Cost profile. X-Part's segmenter occupies a whole four-GPU node, so its 117 s wall clock is charged as 469 GPU seconds. In the 1024 cascade, 51.2 of its 62.4 s go to writing and decimating a mesh of 692K vertices, against 11.1 s of generation. PartPacker is the cheapest method and the weakest on parts; KaiNinja costs 40% more and gains 0.20 F1^0.05_P. AutoPartGen spends 3× the budget because it generates one part after another, and the segment-then-regenerate cascade costs an order of magnitude more than any other method. Whole-object quality separates these methods very little: four of the five baselines sit within 0.032 of one another, and the fifth is 0.035 above the best of them.
• Two-coloring is a heuristic, not a guarantee. It works only when the part-contact graph is bipartite; objects whose parts interlock densely are merged more than their annotation intends. The authors argue that PartPacker's suggested remedy of converting the contact graph to a planar graph and applying the four-color theorem does not hold in general, because a 3D contact graph need not be planar and making it planar deletes contact edges, putting touching parts back into one volume. They state that no fixed number of volumes removes this limit.
• Thin shells sometimes escape the duplicate-occupancy merge step (stated in the paper as a limitation, with details deferred to Section 5).

Methodology in Plain English

The key observation is that TRELLIS.2's O-Voxel representation stores at most one sheet of surface per voxel. Where two parts touch, their faces are two nearly coincident sheets, so at any finite resolution one is kept and the other discarded. This is a capacity limit, not a resolution limit, so no architecture built on a single volume can recover a part interface the input never contained.

The fix is packing, borrowed from PartPacker but moved onto the sparse voxel grid TRELLIS.2 already denoises. Each object's parts form a contact graph: nodes are parts, and two nodes are joined when some voxel carries facets of both, weighted by the number of shared voxels. This graph is two-colored so that the two color classes become two volumes, A and B, in which parts never touch. When the graph is not bipartite, edges are greedily contracted — taking an odd cycle, merging the heaviest-weight edge's two parts, and rebuilding — until it is, then breadth-first search two-colors it.

Because the pretrained prior only understands whole objects voxelized as one volume, the model is trained to move the weights as little as the task requires. The first stage, the layout flow, denoises the coarse occupancy of both volumes. Each stream is a full copy of the pretrained transformer (30 blocks, width 1536, 12 heads, rotary position encoding) with its own input/output projections and timestep modulation, so stream identity lives in the weights. After blocks at depths {6, 12, 18, 24, 29}, one shared cross-volume attention block attends over both streams; these blocks are zero-initialized, so training begins as two independent pretrained denoisers. Training proceeds in phases: fit each volume separately on its own marginal distribution, merge the two fitted copies without loss, freeze everything and warm up only the cross-volume blocks, then fine-tune jointly. A disjointness penalty, the expected element-wise product of the two streams' decoded soft occupancies, is added to the flow-matching loss with weight 5, since nothing in a per-stream loss forbids the streams from claiming the same voxel.

The second stage, the refinement flow, denoises the fine structured latent and keeps the pretrained backbone unchanged. Its two streams coexist by scoping attention: in 20 of the 30 blocks each self-attention layer sees only one volume's tokens, and in the remaining 10 blocks (2, 5, 8, ..., 29) it attends over both. Since self-attention handles a variable token count natively, this regrouping changes no layer width and adds no backbone parameters; stream identity in the merged blocks comes from a learned volume embedding initialized at zero. This stage uses a frozen warmup that updates only the volume embedding, then full fine-tuning.

A frozen DINOv3 ViT-L/16 encoder reads the image at 512×512 and conditions both stages through cross-attention, with classifier-free guidance at inference. The structured latent autoencoder is the pretrained TRELLIS.2 one and stays frozen, so only the two flows are trained.

At inference, two light post-processing steps turn the volumes into parts, neither using a mask or a segmenter. First, duplicate occupancy is merged: for every pair of connected components, one from each volume, if they overlap by more than half and are embedded at least three voxels deep, the smaller component's voxels move into the larger's volume, so nothing is deleted. Second, the frozen TRELLIS.2 VAE decodes each volume into an O-Voxel volume and a textured mesh, and connected components of each mesh become parts; components whose surfaces lie within two fine voxels of each other are treated as one part, and fragments below 0.5% of the volume's surface area are attached to the nearest part.

Training data come from one corpus of 19,132 objects assembled from four datasets: Articraft-10K (8,968), PartNext (6,358), TRELLIS (1,931), and Fusion360 (1,875). A test set of 1,000 objects spans all four sources (Articraft 513, PartNext 297, Fusion360 102, TRELLIS 88). OmniPart produces no output on 14 of the 1,000 objects because its sparse convolution backend runs out of resources on very dense voxel grids, so all methods are reported on the 986 objects every method completes. Metrics follow OmniPart: shapes are normalized into [-0.5, 0.5]³, and Chamfer distance and F-score at thresholds 0.1 and 0.05 are reported for the whole object (W) and parts (P). Predictions are aligned to ground truth by picking, out of 48 signed axis permutations, the one with the smallest whole-object CD. Ground-truth and predicted parts are matched one-to-one by Hungarian assignment at cost 1 − IoU on nearest-neighbor-transferred surface labels, with unmatched predictions and uncovered ground-truth parts both penalized. The paper states that OmniPart does not specify its matching protocol and no compared method releases part-metric code, so this protocol is the authors' own choice.

Why This Matters

Impact on research. The paper makes a representation-level claim rather than only a systems claim: that a single volume cannot hold a part interface at any resolution, and that packing touching parts into separate volumes restores one sheet per voxel. If packing also improves whole-object fidelity, as the reported 0.919 F1^0.05_W against 0.830 for the fine-tuned backbone suggests, that reframes dual-volume packing as a better representation of the same objects rather than only a container for parts. It also demonstrates a reusable recipe for extending a pretrained whole-object generator into a new output structure while preserving its speed and quality, and it opens agent-authored assets as a supervised data source with exact part labels by construction.

Real-world applications:

• Game and film asset pipelines, where each part must be separately movable and retexturable.
• Rigging and animation, since every generated part is a self-contained sub-mesh that can be moved, retextured, or rigged on its own.
• Simulation and robotics, where physical interaction operates on parts rather than fused geometry.
• Interactive 3D content and e-commerce, where a single RGB image and roughly 24 seconds on one H100 yields a part-separated asset.

Industry relevance. The cost comparison is framed around deployability: the strongest segment-then-regenerate cascade runs an order of magnitude slower than in-generation methods and needs a segmenter occupying a whole four-GPU node, while KaiNinja runs the backbone once for any number of parts with no upper bound on that number and no segmenter on the critical path. Removing the mask or box also removes a dependence on segmentation accuracy that the authors identify as an uncorrectable failure mode of mask-dependent methods.

Future Directions

• Handling non-bipartite contact graphs. Two-coloring is a heuristic that merges densely interlocking parts more than annotation intends. The authors argue that a fixed number of volumes cannot remove this limit, so what does is an open question raised by the paper rather than answered by it.
• Recovering thin shells that escape the duplicate-occupancy merge. The paper notes thin shells sometimes evade the between-stage merge step and points to Section 5 for a partial resolution, so a more robust merge remains an open problem.
• Other conditioning signals. The paper states that nothing in either stage is specific to images, so conditioning signals other than a single RGB image could be substituted.
• Comparison against explicit-connectivity mesh generators. LATO.2 and Meshy T2 are described as complementary, but the part-capable variant of LATO.2 has no released weights and Meshy T2 has announced a code and weight release, so a direct comparison is deferred. The authors also flag explicit connectivity as hardest exactly where part structure is decided, at dense contacts and at interior surfaces the input view never shows.
• Scaling behavior with many parts. The paper claims no upper bound on N and that the backbone runs once, but it does not report how part-level metrics change as the number of parts grows.

Target Audience

Researchers and practitioners in 3D generative modeling and computer graphics who work on image-to-3D generation, part-aware or compositional generation, or 3D asset pipelines. It is most useful to readers who already understand diffusion or rectified flow models, sparse voxel latents, and existing generators such as TRELLIS, TRELLIS.2, and Hunyuan3D, and to engineers building production asset pipelines who need part-separated output at interactive cost. Readers looking for a gentle introduction to 3D generation will find the representation-level argument accessible but the architecture and training schedule demanding.

Authors’ abstract

Native 3D generators turn one image into a single mesh. TRELLIS.2 and its peers deliver high-fidelity non-watertight geometry with materials, but the output is one fused object, while downstream work such as editing, rigging and simulation operates on part-level assets. A naive idea is to run a 3D segmentation network on the fused mesh that TRELLIS.2 generates, but such pipelines are slow and bounded by the accuracy of the segmentation. We want a simple way to extend an existing native 3D generator to the part level. But we face a critical problem: the O-Voxel grid stores one sheet of surface per voxel, so a single volume cannot represent the interface where two parts touch, at any resolution. We introduce a dual-volume representation to solve this problem and put forward KaiNinja, a part-level extension of TRELLIS.2 built on a dual-volume form of its O-Voxel representation. KaiNinja keeps the generation speed and quality of TRELLIS.2 while extending it to the part level, with no mask or segmenter in the pipeline. Its training data come from sources of many kinds, including CAD models and assets authored by an LLM-driven agent; to our knowledge it is the first 3D generative model trained on agent-authored part data. Surprisingly, we also find that whole-object fidelity improves over the same backbone fine-tuned on the same dataset. Against part generation pipelines of different paradigms, it lowers whole-object Chamfer distance by 40% and raises strict part F-score by 16%.