ShapeUP: Scalable Image-Conditioned 3D Editing

AI summary

Overview

Research area: Computer vision and 3D generative modeling, specifically image-conditioned editing of 3D assets built on native 3D foundation models.

Technical level: Advanced. The paper assumes familiarity with diffusion transformers, latent representations, LoRA adaptation, classifier-free guidance, and multi-view texture baking.

Scope: The paper introduces ShapeUP, a framework that recasts 3D asset editing as supervised latent-to-latent translation inside the native latent space of a pretrained image-to-3D foundation model (Step1X-3D), conditioned jointly on the source shape and a single edited 2D image.

What This Paper Is About

Existing ways of editing 3D assets force a trade-off: optimization-based methods are accurate but prohibitively slow, multi-view 2D propagation methods suffer from visual drift and registration artifacts, and training-free latent manipulation methods are bound by the fixed, frozen priors of their pretrained weights and cannot improve by scaling up data. ShapeUP's goal is to edit an existing 3D shape (changing pose, deformation, parts, or material) so that it matches a user-supplied edited image, while strictly preserving the identity and structure of the original asset and localizing the change implicitly without hand-drawn masks.

Key Contributions

1. A supervised latent-to-latent reformulation of 3D editing. Instead of optimization or multi-view 2D propagation, ShapeUP trains a direct mapping inside the native 3D latent space of a pretrained 3D foundation model, so the model can benefit from scaling rather than being limited by frozen priors.

2. An image-as-prompt conditioning design. Both the source shape (encoded into latent tokens) and a single edited 2D image are fed to a 3D Diffusion Transformer (DiT) with LoRA adapters, giving fine-grained visual control, implicit mask-free localization, and preserved structure.

3. A synthetic supervision dataset spanning local and global edits. The dataset contains Parts samples (progressive component removal) and Distant Frames in Motion (DFM) samples (temporally distant keyframes from animation sequences) that capture pose and deformation changes while preserving object identity.

4. A two-stage geometry-and-texture editing pipeline plus a new benchmark. Geometry is edited in latent 3D space; texture is synthesized by a fine-tuned multi-view model that is explicitly conditioned on the source texture. The authors also introduce BenchUp, a benchmark of 24 diverse meshes and 100 edit conditions covering Parts, Global-Deformation, Global-Pose Change, and Global-Texture/Material edits.

Main Findings

• ShapeUP outperforms both baselines on BenchUp across every reported metric. On Condition Alignment, ShapeUP scores SSIM 0.763, LPIPS 0.198, CLIP-I 0.943, DINO-I 0.915, and CLIP-Dir 0.520, versus 3DEditFormer (0.733 / 0.270 / 0.908 / 0.849 / 0.441) and EditP23 (0.759 / 0.254 / 0.917 / 0.851 / 0.455). On Occluded Region Fidelity, ShapeUP scores CLIP-I 0.928 and DINO-I 0.878, versus 3DEditFormer (0.877 / 0.736) and EditP23 (0.880 / 0.748).

• The method resolves the usual fidelity-versus-consistency trade-off. The authors state that ShapeUP achieves superior condition alignment without sacrificing preservation of unedited regions, unlike baselines that hallucinate inconsistent textures or degrade geometry in regions hidden from the conditioning image.

• Users strongly preferred ShapeUP in a two-alternative forced choice study. The study collected 664 comparisons from 34 participants; the authors report that ShapeUP was strongly preferred over each baseline, with 95% confidence intervals shown in the paper.

• DFM data is critical for global edits but trades off against occluded-region scores. Ablating DFM ("w/o Motion") raises Occluded Region Fidelity to CLIP-I 0.932 and DINO-I 0.884 (above the full model's 0.928 / 0.878) but reduces CLIP-Dir to 0.505, and qualitatively produces noticeably degraded pose and global edit behavior. The authors attribute the higher occluded-region scores to Parts-only supervision, which forces adherence to the original layout at the cost of flexibility.

• More source-shape latents help. Training with 256 latents gives SSIM 0.766 / LPIPS 0.206 / CLIP-I 0.941 / DINO-I 0.909 / CLIP-Dir 0.525, and 512 latents gives 0.768 / 0.220 / 0.918 / 0.868 / 0.506, versus the 1024-latent configuration used in all other experiments (0.763 / 0.198 / 0.943 / 0.915 / 0.520). The 1024 setting gives the strongest DINO-I and the mixed picture is discussed as the best overall performance.

• Concatenating source multi-view images into the UNet is a worse texture strategy. The "Concat MV" variant achieves the highest Condition Alignment scores (SSIM 0.779, C-Dir 0.555) but drops to CLIP-I 0.897 and DINO-I 0.832 in occluded regions, whereas the paper's cross-attention injection of source views gives the better trade-off (0.928 / 0.878).

• Classifier-free guidance scales act as a tunable dial. For geometry, the paper uses image guidance 2.5 and shape guidance 3.5 at inference; for texture, edited-image guidance 2.5 and multi-view guidance 3.5. The paper shows that varying these (for example s_i = 6.5 / s_mv = 2.5 through s_i = 2.5 / s_mv = 6.5) moves results along a controllability-versus-preservation axis.

• Reconstruction fidelity is also strong in a Zero-Edit experiment. With identity edits, ShapeUP reaches visible-region SSIM 0.821, LPIPS 0.124, CLIP-I 0.951, DINO-I 0.928 and occluded-region SSIM 0.793, LPIPS 0.135, CLIP-I 0.934, DINO-I 0.874, versus 3DEditFormer (0.780 / 0.221 / 0.929 / 0.890 visible; 0.784 / 0.221 / 0.904 / 0.798 occluded) and EditP23 (0.800 / 0.216 / 0.921 / 0.867 visible; 0.824 / 0.213 / 0.894 / 0.802 occluded). EditP23 leads on occluded-region SSIM in this particular test.

• Known failure modes are tied to the frozen VAE. ShapeUP degrades on assets with fine-grained geometric detail or non-object-centric compositions because all shapes pass through the pretrained Step1X-3D VAE, whose encoder–decoder smooths thin structures. The authors show a VAE-only reconstruction to argue that much of the degradation comes from the latent space's limited expressiveness rather than the editing process. Inference speed and parameter counts are not reported in the provided content.

Methodology in Plain English

ShapeUP borrows the two-stage structure of the Step1X-3D image-to-3D foundation model and fine-tunes each stage for editing.

Geometry stage. The source mesh is encoded by a pretrained shape VAE into 2048 latent vectors, which are subsampled to 1024 latent tokens as the source-shape condition. The edited image is encoded with CLIP and DINOv2 and concatenated into the same conditioning sequence. A 3D Diffusion Transformer (a FLUX-inspired MMDiT with alternating double-stream and single-stream blocks) is then trained with LoRA adapters on both block types to translate the source latent into the edited target latent. Training uses triplets: a source shape, an edited 2D image, and the corresponding edited 3D shape. Because the model is conditioned on the source shape and an image rather than a spatial mask, localization of the edit is implicit.

Texture stage. The edited geometry is rendered to normal and position maps. A pretrained single-view-to-multi-view diffusion model (built on the MV-Adapter design) generates consistent multi-view images of the edited shape conditioned on the edited reference image and on multi-view renders of the source texture. Source-view features are injected through the existing image cross-attention layers, distinguished from the target-edit tokens by a view-axis positional encoding. The resulting views are baked onto the edited geometry.

Training data. The dataset contains 7,430 textured meshes from Objaverse: 560 are Distant Frames in Motion (DFM) sequences and the remainder are Parts samples. Parts samples were selected from a 30k filtered asset list used to train Step1X-3D's texture model, keeping 6,870 distinct assets with at least two geometry nodes, then generating 2–4 progressive component-removal variants. DFM assets come from the Objaverse Animation subset, restricted to those also in the Step1X-3D texture list, with three keyframes selected per asset. Because of class imbalance, DFM samples were upsampled threefold, giving an effective DFM sampling probability of 22%.

Training configuration. Geometry: LoRA adapters of rank 128 on a single NVIDIA L40S GPU, batch size 8, 500K iterations, image dropout 0.2 and shape dropout 0.1. Texture: Step1X-3D UNet adapter layers fine-tuned on an NVIDIA A100 PCI GPU, batch size 4, 27K iterations. Both pipelines use the same objectives as their base models.

Evaluation. BenchUp contains 24 diverse meshes and 100 edit conditions built from Objaverse geometries augmented with TRELLIS 2.0 and Hunyuan 3d 2.0 meshes, with edit instructions generated by Gemini 3 Pro (Image Preview) across four edit categories and 20 rendered views per mesh (azimuth in [−30°, 30°]). Metrics are split into Condition Alignment (SSIM, LPIPS, CLIP-I, DINO-I, CLIP-Dir) and Occluded Region Fidelity (CLIP-I, DINO-I).

Why This Matters

Impact on research. The paper argues that 3D editing should be a learned, scalable task rather than an optimization or inference-time propagation trick. By conditioning a 3D foundation model on a compact shape latent code instead of deep features from a frozen model (as 3DEditFormer does), it offers a structured, geometry-faithful conditioning signal. It also establishes a benchmark for global 3D edits, addressing what the authors describe as an evaluation gap in prior work that focused on localized edits or narrow mesh diversity.

Real-world applications:

• Game and film asset pipelines where an existing prop or character must be re-posed, deformed, or restyled while keeping its identity intact.
• E-commerce and product visualization, generating variants of a 3D product from a single reference image edit.
• Character animation workflows, since DFM supervision lets the model follow pose changes drawn from motion sequences.
• Iterative content creation, using the paper's multi-step editing setup where edits are applied sequentially and each stage is fed the original shape's multi-view renders to avoid color drift.

Industry relevance. The method is backbone-agnostic and relies on lightweight LoRA adaptation of an existing foundation model, meaning teams with an existing image-to-3D backbone can add editing capability without training from scratch. The feed-forward, mask-free formulation is directly relevant to interactive tools where manual mask painting and per-edit optimization are impractical.

Future Directions

• Scaling the supervision data. The authors identify the current training set as relatively small and biased toward mostly closed, object-centric assets, and list larger and more diverse training data as a promising direction.
• Breaking the VAE representational ceiling. Since output quality is bounded by the pretrained shape VAE's capacity, improving latent expressiveness or adding detail-recovery mechanisms could address thin structures and intricate geometry.
• Handling non-object-centric compositions. The current formulation assumes object-centric assets; extending to scenes or compositions with fine-grained detail is left open.
• Exploring alternative conditioning and injection designs. The ablations on latent count (256 / 512 / 1024), DFM inclusion, and source multi-view injection (cross-attention versus channel concatenation) leave room for further variations, and the authors frame the broader question of learnable native-3D editing frameworks as an open direction.

Target Audience

Researchers and practitioners in 3D generative modeling and graphics who are already comfortable with diffusion models, latent representations, and foundation-model fine-tuning. It is most useful for those building 3D content-creation tools, evaluating 3D editing methods, or designing supervision datasets for 3D tasks. Readers looking for an introductory treatment of 3D diffusion or for published inference-time and model-size numbers will find this paper oriented toward method design, benchmark construction, and quality metrics instead.

Authors’ abstract

Recent advancements in 3D foundation models have enabled the generation of high-fidelity assets, yet precise 3D manipulation remains a significant challenge. Existing 3D editing frameworks often face a difficult trade-off between visual controllability, geometric consistency, and scalability. Specifically, optimization-based methods are prohibitively slow, multi-view 2D propagation techniques suffer from visual drift, and training-free latent manipulation methods are inherently bound by frozen priors and cannot directly benefit from scaling. In this work, we present ShapeUP, a scalable, image-conditioned 3D editing framework that formulates editing as a supervised latent-to-latent translation within a native 3D representation. This formulation allows ShapeUP to build on a pretrained 3D foundation model, leveraging its strong generative prior while adapting it to editing through supervised training. In practice, ShapeUP is trained on triplets consisting of a source 3D shape, an edited 2D image, and the corresponding edited 3D shape, and learns a direct mapping using a 3D Diffusion Transformer (DiT). This image-as-prompt approach enables fine-grained visual control over both local and global edits and achieves implicit, mask-free localization, while maintaining strict structural consistency with the original asset. Our extensive evaluations demonstrate that ShapeUP consistently outperforms current trained and training-free baselines in both identity preservation and edit fidelity, offering a robust and scalable paradigm for native 3D content creation.