VecSet-Edit: Unleashing Pre-trained LRM for Mesh Editing from Single Image

AI summary

Overview

• Research area: Computer Vision / 3D graphics — localized 3D mesh editing guided by a single 2D image, built on top of a pre-trained VecSet Large Reconstruction Model (LRM).
• Technical level: Advanced. The paper assumes familiarity with diffusion transformers, latent 3D representations (VecSet, voxels, triplanes), cross- and self-attention maps, rectified flow, and mesh-based rendering (SDF decoding, Marching Cubes, UV texture baking).
• Scope: The paper introduces a training-free pipeline, VecSet-Edit, that performs localized geometry and texture edits on an existing 3D mesh using one rendered view plus a 2D edit mask, while preserving the mesh outside the edit region.

What This Paper Is About

Most 3D editing methods operate on intermediate representations such as 3D Gaussian Splatting or multi-view images, not directly on meshes with explicit topology, and prior mesh-editing work such as VoxHammer relies on voxel grids that cap resolution and require labor-intensive 3D mask annotation. This paper asks whether the high-fidelity VecSet LRM (TripoSG) can be repurposed for mesh editing without any retraining. The goal is to take a reference mesh, a rendered view, a user-edited target image, and a binary 2D mask, and produce an edited mesh that follows the requested semantic change while faithfully preserving the original geometry and texture elsewhere.

Key Contributions

1. VecSet-Edit framework: The first training-free framework that performs localized 3D mesh editing directly in the latent VecSet space of a high-fidelity reconstruction model, avoiding the resolution bottleneck of voxel-based LRMs.
2. Two-stage token localization: Mask-guided Token Seeding derives a coarse editable token set from a 2D mask, and Attention-aligned Token Gating then retains tokens with the strongest spatial correlation to the target region, preventing edits from leaking into unintended areas.
3. Drift-aware Token Pruning: Removes drifted tokens that would otherwise break geometric consistency during the denoising process, keeping boundaries clean.
4. Detail-preserving Texture Baking: Updates textures only where geometry changes, preserving appearance details in untouched regions.

The paper also formally states and empirically verifies the VecSet Geometry Property, showing that unordered VecSet tokens nonetheless exhibit spatial locality and can be partitioned into editable and preserved subsets.

Main Findings

• Tokens have spatial locality: On 300 mesh/bounding-box pairs from Edit3D-Bench, geometry decoded purely from a bounding-box-restricted token subset Decode(V_B) achieved a Chamfer Distance below ε = 0.30 in 82.3% of samples, below ε = 0.10 in 73.5%, below ε = 0.05 in 69.4%, and below ε = 0.01 in 44.2%. The reference Decode(V) ∩ B reached 100% for ε = 0.30, 0.10 and 0.05, and 98.6% for ε = 0.01.
• Best preservation among editing baselines: On Edit3D-Bench, VecSet-Edit reached CD 0.011, PSNR 29.63, SSIM 0.97, and LPIPS 0.04 on unedited regions, versus VoxHammer (CD 0.018, PSNR 27.05, SSIM 0.95, LPIPS 0.05), Trellis (CD 0.014, PSNR 29.22, SSIM 0.97, LPIPS 0.04), MVEdit (CD 0.188, PSNR 21.90, SSIM 0.91, LPIPS 0.13), and Instant3DiT (CD 0.124, PSNR 16.76, SSIM 0.81, LPIPS 0.28). The paper reports this CD as 21% lower than the previous state of the art.
• Condition alignment trade-off: VecSet-Edit recorded the highest DINO-I at 0.92, while its CLIP-T of 27.75 was slightly below Trellis (27.87) and VoxHammer (28.08). The authors describe this as an intentional design priority favoring strict visual alignment over broad textual correspondence.
• Faster than the strongest baselines: VecSet-Edit ran in approximately 200 s, about a 2× wall-clock speedup relative to Trellis and VoxHammer (both approximately 600 s). MVEdit took approximately 160 s, Instant3DiT approximately 20 s, and the TripoSG VAE encode-decode upper bound approximately 100 s.
• Upper bound reference: The TripoSG backbone's own VAE encode-decode, listed as the upper bound for preservation quality, achieved CD 0.006, PSNR 31.88, SSIM 0.98, LPIPS 0.02, and FID 16.21.
• User study preference: With 12 3D practitioners evaluating 10 editing cases (120 total votes per criterion), VecSet-Edit won 59.17% for Region Preservation (VoxHammer 21.67%, MVEdit 10.83%, I3DiT 8.33%) and 58.33% for Condition Alignment (VoxHammer 26.67%, MVEdit 14.17%, I3DiT 0.83%).
• Ablation confirms each component: Starting from RePaint alone (CD 0.024, PSNR 24.35, LPIPS 0.07, DINO-I 0.88, CLIP-T 27.25), adding Token Seeding improved CD to 0.006, adding Token Gating recovered PSNR/LPIPS to 24.71/0.06, adding Token Pruning raised DINO-I to 0.89, and the full pipeline with Detail-Preserving Texture Baking reached PSNR 29.63, LPIPS 0.04, DINO-I 0.92, CLIP-T 27.75.
• No 3D mask needed at inference: Unlike baselines that rely on the 3D bounding box for localization, VecSet-Edit uses the box only for evaluation; localization comes from a 2D image mask.

Methodology in Plain English

The team starts from TripoSG, an LRM that represents a 3D shape as a set of latent vectors ("tokens") rather than as a grid. A VAE encodes a surface point cloud into these tokens and decodes tokens back into a mesh via a signed distance field and Marching Cubes; a diffusion transformer then generates tokens conditioned on image features from a frozen DINOv2 encoder.

The key insight is that, even though the tokens are stored as an unordered set, individual tokens or small groups of them consistently correspond to coherent surface regions. The authors verify this by cropping a 3D box, gathering the tokens whose query points fall inside the box, decoding them alone, and measuring Chamfer Distance against the cropped reference geometry.

Based on this, editing becomes a token-selection problem. Token Seeding looks at the cross-attention maps between tokens and image pixels, averages the attention mass landing inside the 2D mask, and keeps tokens above a threshold; only attention layers with high KL divergence are used, since those correlate with strong condition alignment. Token Gating then uses self-attention, which is more stable across layers, to pull in additional tokens that are strongly coupled to the seeded region.

With editable tokens V_E and preserved tokens V_P separated, the method adapts RePaint to VecSet space: editable tokens are denoised toward the target image while preserved tokens are pinned to their original diffusion trajectory, then recombined at each step. Because VecSet tokens move freely in space rather than sitting on a fixed grid, some editable tokens can drift into preserved regions early in denoising. Drift-aware Token Pruning intervenes at a chosen timestep, removing tokens that are structurally associated with the preserved region unless they are also supported by the target image condition, so genuine edits survive while conflicts are discarded.

Finally, for appearance, the pipeline uses MV-Adapter to generate 6 consistent multi-view RGB images from rendered surface normals plus the target image, and projects them onto the mesh via differentiable rendering. To avoid re-baking the whole mesh, the team computes geometric difference masks from normal-map changes between the original and edited meshes and restricts generation to those regions only.

Settings reported in the main text: RePaint starting timestep T_repaint = 0.7 and pruning timestep T_pruning = 0.6, with remaining settings and threshold analysis in the appendix.

Why This Matters

The work closes a gap between 3D generative models and practical, controllable asset production: it shows that a pre-trained reconstruction backbone can be steered for precise local edits without any fine-tuning, without 3D supervision, and without the resolution ceiling of voxel representations. This is relevant to research on latent-space controllability, since it demonstrates that unordered latent token sets carry exploitable spatial structure.

Real-world applications:

• Game and film asset pipelines: Artists can adjust a specific part of a generated asset (for example, swapping a car's wheels or changing a character's head) while keeping the rest of the model intact, instead of regenerating from scratch.
• Animation and simulation: Because the output remains a mesh with explicit topology, edits remain usable for rigging, physics-based simulation, and downstream tooling that cannot consume Gaussian splats or radiance fields.
• E-commerce and product visualization: Rapid variant creation, such as changing a product component or color region, with the rest of the model's appearance preserved.
• AR/VR and 3D printing: Localized mesh refinement for assets that must stay geometrically consistent with an original reference.

Industry relevance centers on cost and speed: the method is training-free, uses only a 2D mask instead of labor-intensive 3D annotation, and runs in roughly 200 s, about 2× faster than the strongest baselines it compares against.

Future Directions

• More robust token selection or refinement: The authors state that reconstruction fidelity from naive token subsets should improve with better selection strategies, which would raise the ceiling on edit precision.
• Tight-tolerance reconstruction remains hard: Only 44.2% of subset-decoded samples reached ε = 0.01, so improving fine-grained geometric fidelity at strict tolerances is an open problem.
• Balancing visual versus textual alignment: VecSet-Edit leads on DINO-I but trails slightly on CLIP-T, leaving open how to satisfy image alignment and broader semantic/textual correspondence simultaneously.
• Generalizing beyond the current setup: The framework is demonstrated with TripoSG as backbone and a user-provided 2D mask and edited target image; extending to other VecSet backbones, other single-view conditioning, or automatically derived masks are natural next steps implied by the design. These directions are not explicitly enumerated in the paper.

Target Audience

This paper suits researchers and graduate students in 3D vision, graphics, and generative modeling who work on 3D editing, latent diffusion models, or large reconstruction models, as well as technical artists and graphics engineers evaluating whether LRM-based editing can fit into production pipelines. Readers need a working understanding of attention mechanisms, diffusion denoising, and mesh representations (SDF, Marching Cubes, UV texture maps) to follow the methodology; the core intuition about token locality is accessible to a broader audience, but the implementation details are specialized.

Authors’ abstract

3D editing has emerged as a critical research area to provide users with flexible control over 3D assets. While current editing approaches predominantly focus on 3D Gaussian Splatting or multi-view images, the direct editing of 3D meshes remains underexplored. Prior attempts, such as VoxHammer, rely on voxel-based representations that suffer from limited resolution and necessitate labor-intensive 3D mask. To address these limitations, we propose \textbf{VecSet-Edit}, the first pipeline that leverages the high-fidelity VecSet Large Reconstruction Model (LRM) as a backbone for mesh editing. Our approach is grounded on a analysis of the spatial properties in VecSet tokens, revealing that token subsets govern distinct geometric regions. Based on this insight, we introduce Mask-guided Token Seeding and Attention-aligned Token Gating strategies to precisely localize target regions using only 2D image conditions. Also, considering the difference between VecSet diffusion process versus voxel we design a Drift-aware Token Pruning to reject geometric outliers during the denoising process. Finally, our Detail-preserving Texture Baking module ensures that we not only preserve the geometric details of original mesh but also the textural information. More details can be found in our project page: https://github.com/BlueDyee/VecSet-Edit/tree/main