SeMo3D: Unifying 3D Generation, Editing, and Completion via Selective Modeling

AI summary

Overview

Research area: Generative 3D computer vision — native 3D diffusion models for asset generation, editing, and completion.

Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching models, transformer architectures, structured latent representations, and 3D asset pipelines.

Scope: SeMo3D proposes a single paradigm — "selective modeling" — that recasts 3D generation, editing, and completion as synthesizing only a designated region of a 3D asset while leaving the rest untouched, and instantiates it with a region-selective denoising objective and a multi-condition 3D diffusion transformer (Omni3DiT).

What This Paper Is About

Existing 3D methods treat generation, editing, and completion as separate problems that all regenerate a complete 3D asset under different constraints, even when only a small local region needs to change. This causes three problems the authors identify: task isolation (separate models per task), region inconsistency (parts that should stay the same get altered), and dependence on large paired editing/completion datasets. The paper's goal is one model that handles all three tasks by synthesizing only the selected region — the full asset for generation, the target region for editing, the missing region for completion — while reusing the rest of the asset unchanged.

Key Contributions

1. Selective modeling as a unifying paradigm. The authors formulate 3D generation, editing, and completion as a single problem: given tokens partitioned into a selected set S (to be synthesized) and a context set C (to be preserved), denoise only S. The three tasks differ solely in how that partition is defined.
2. Region-selective denoising. A training procedure that randomly selects regions of varying sizes on ordinary 3D assets, adds noise and computes loss only on the selected tokens, and thereby learns all three tasks from standard 3D data — no paired editing or completion supervision.
3. Omni3DiT. A multi-condition 3D diffusion transformer that jointly processes clean context tokens and noisy selected tokens, distinguishing them with learned role embeddings and fusing image, text, and sparse point guidance.
4. Empirical validation plus an agentic workflow. Experiments on three benchmarks (Toys4K, Edit3D-Bench, Omni-Comp) show gains over specialized state-of-the-art methods on all three tasks, and the model is shown to support natural-language-driven 3D creation when integrated with an agent.

Main Findings

• Generation (Toys4K): SeMo3D with image, text, and point conditions achieves the best LPIPS (0.079), FID (4.99), DINO similarity (0.926), and Chamfer Distance (0.0118), with the second-best KID×100 (0.059, behind Trellis2's 0.057). It beats image-only, text-only, and dual-condition baselines including Trellis, ShapeLLM-Omni, HY3D-2.1, Trellis2, Pixal3D, TIGON, and HY3D-Omni.
• Editing (Edit3D-Bench): SeMo3D leads all three metric groups — consistency (CD 0.0088, PSNR 37.39, SSIM 0.986, LPIPS 0.028), quality (FID 22.46, FVD 113.6), and accuracy (DINO 0.875, CLIP 0.286) — without any paired editing data, against training-free (VoxHammer, Nano3D) and training-based (3DEditFormer, PartFlow) baselines.
• Completion (Omni-Comp): Even without point guidance, SeMo3D achieves the best CD (0.0208), EMD (0.0346), and F-Score (0.468); adding sparse points improves these to 0.0133, 0.0223, and 0.541, despite training on no paired completion data.
• Complementary modalities: Ablation shows text mainly improves perceptual quality and semantic fidelity (FID, DINO), points mainly improve geometry (CD, EMD, F-Score), and the combination is best overall. With a fixed image, varying text and points yields different objects (sofa, suitcase, basket, ottoman; or tractor, rover, truck, ATV).
• Role embedding matters for context distinction: Removing it barely changes generation (where all tokens are selected) but degrades editing (CD 0.0156 vs 0.0133) and completion (EMD 0.0249 vs 0.0223, F-Score 0.528 vs 0.541).
• Point injection method matters: Injecting points via cross-attention instead of self-attention doubles generation CD (0.0244 vs 0.0118) and degrades completion.
• Dropout strategy matters: Replacing compositional condition dropout with standard CFG dropout weakens text and point guidance, with metrics nearly reverting to the image-only configuration (LPIPS 0.110 vs 0.079, CD 0.0331 vs 0.0118).
• Robustness: Edits remain nearly identical when the edit mask is enlarged by up to 40% about its center; generation stays consistent across 16, 32, and 48 condition points (the model was trained with at most 32); and results hold across different point spatial distributions.
• Efficiency and agent use: Generating a 512-resolution asset takes about 5 seconds, on par with Trellis2. Compared with direct Blender coding, the same agent using SeMo3D achieves higher modeling fidelity with lower execution time and agent cost.
• Failure cases: The inherited structured latents cannot capture very thin structures such as palm fronds, and the CLIP text encoder struggles with compositional prompts — orange fruits are rendered as orange texture rather than as separate objects.

Methodology in Plain English

The model builds on Trellis2's representation of a 3D asset as spatially structured latent tokens produced in three stages: a sparse-structure stage on a 16³ grid that decides which voxels are occupied, then shape and texture stages that produce tokens only at occupied positions on a 32³ grid. Because each token corresponds to a localized region of space, the representation supports spatially selective operations.

The central trick is to split tokens into two sets: a selected set that gets denoised and a context set that stays clean. At training time, the method picks a random region — sometimes the whole asset, sometimes a random 3D box — adds noise only to tokens in that region, feeds the model the mixed input plus a binary mask, and computes loss only on the selected tokens. Training samples either the full-region mode or the local-region mode with equal probability, so a single objective spans generation, editing, and completion. No paired before/after data is needed, because any ordinary asset can be split into a region and its surroundings.

The architecture, Omni3DiT, extends an image-conditioned 3D diffusion transformer with four additions: (1) learned role embeddings that tell the model which tokens are noisy versus clean; (2) a text cross-attention branch using CLIP features, in parallel with the existing image branch; (3) sparse points injected into the token sequence and positioned with RoPE, so self-attention can reason about their spatial relationship to the latent tokens; and (4) compositional condition dropout, which withholds random condition subsets — the image is dropped in 30% of steps — so the text and point branches cannot simply lean on the pretrained image pathway. New modules are zero-initialized to preserve pretrained behavior at initialization.

At inference, the user defines the partition: the whole asset for generation, a target mask for editing, or the complement of a partial input's occupancy for completion. Selected tokens start from noise and are denoised while context tokens are reused unchanged at every step, which preserves unedited content by construction. The three stages run sequentially, with the sparse-structure partition transferred to the finer grid by nearest-neighbor upsampling. Training uses Trellis-500K at 512 resolution with a learning rate of 5×10⁻⁵ and batch size 32 (200K steps for the sparse-structure stage, 100K for each SLat stage), initialized from Trellis2.

Why This Matters

The work argues that a single model, trained without task-specific paired datasets, can replace separate specialized pipelines for three of the most common 3D creation operations. Because unchanged tokens are never regenerated, source fidelity becomes a structural guarantee rather than a soft training objective — a meaningful shift from the "regenerate everything and hope it matches" approach that the authors show leads to visible drift in unedited regions. The observation that editing and completion can be learned purely from ordinary 3D assets also lowers the data barrier for future work.

Real-world applications the paper points to:

• Games: rapid creation and modification of props and assets through natural-language requests.
• AR/VR: iterative asset authoring where only part of an object needs to change.
• Robotics: building and repairing 3D models of environments and objects.
• Digital twins: completing and updating scanned geometry, where partial scans are the norm and preserving measured regions is essential.

Industry relevance: The agentic demonstration (GPT-6 Astra with Codex invoking SeMo3D versus writing Blender code directly) positions the method as a backend for natural-language 3D authoring tools, where fidelity, speed (about 5 seconds per 512-resolution asset), and the guarantee that untouched geometry stays untouched are all commercially valuable. Avoiding 100K-scale paired editing datasets also reduces the cost of building such systems.

Future Directions

• Higher-fidelity 3D representations. The authors note that inherited structured latents cannot capture very fine structures such as thin palm fronds, and suggest higher-fidelity representations as a remedy.
• Stronger vision-language encoders. The CLIP text encoder fails on complex compositional prompts (orange fruits becoming orange texture); stronger encoders are proposed as the fix.
• Beyond local operations. The paper explicitly scopes "editing" to local addition, deletion, and replacement, excluding global operations such as style transfer — extending selective modeling to global edits is an open question.
• Richer conditioning and agent workflows. The demonstrated agent pipeline is a first step; questions remain about how far multi-turn, natural-language-driven 3D creation can be pushed, and whether additional condition modalities or tighter agent-model integration improve results.

Target Audience

Researchers and practitioners working on 3D generative models, diffusion/flow-matching architectures, and multimodal conditioning will get the most from this paper. It is also relevant to graphics and XR engineers evaluating unified 3D authoring backends, and to applied AI teams building agent-driven content creation tools. Readers should already be comfortable with latent diffusion, transformer conditioning mechanisms, and 3D asset representations; the paper is not an introductory survey.

Authors’ abstract

3D generation, editing, and completion are typically treated as regenerating the complete 3D asset under task-specific constraints, even when only a local region needs modification. This leads to task isolation, inconsistency in content that should remain unchanged, and dependence on task-specific paired data. We address these issues with a key observation: the three tasks all synthesize a designated region while keeping the rest intact---the full region for generation, the target region for editing, and the missing region for completion. We term this paradigm Selective Modeling and build SeMo3D upon it; it naturally unifies the three tasks, preserves unchanged content by construction, and enables training without paired editing or completion data, since any 3D asset can be partitioned into a selected region and its context. For training, we propose region-selective denoising, which randomly selects regions of varying sizes on standard 3D assets and applies noise and loss only to them, jointly learning all three tasks. For architecture, we propose Omni3DiT, a multi-condition 3D diffusion transformer that distinguishes regions to create from those to retain and fuses image, text, and point guidance for fine-grained control. Experiments across three benchmarks show SeMo3D surpasses state-of-the-art methods on all tasks, delivering multimodal controllability in generation and strong consistency and accuracy in editing and completion. SeMo3D further serves as a unified modeling tool for agents, enabling efficient 3D creation through natural-language interaction. Project page: https://xiaokunsun.github.io/SeMo3D.github.io/.