3D-Consistent Multi-View Editing by Correspondence Guidance

AI summary

Overview

• Research area: Computer vision, specifically text-based 2D image editing and 3D scene editing of neural representations (NeRFs and 3D Gaussian splats).
• Technical level: Intermediate (readers should be comfortable with diffusion/flow-matching models, denoising guidance, and Gaussian splatting).
• Scope: The paper proposes a training-free guidance framework that makes independently edited views of the same scene agree with one another, and demonstrates it by editing 3D Gaussian splat models from both dense and sparse image sets.

Authors: Josef Bengtson, David Nilsson, Yaroslava Lochman, Fredrik Kahl (Chalmers University of Technology) and Dong In Lee (Korea University). arXiv:2511.22228v3 [cs.CV].

What This Paper Is About

Image editing tools powered by diffusion and flow models work well on a single photo, but if you apply them separately to many photos of the same object or scene, each image gets edited differently — a tie pattern changes shape, a decoration appears in one view but not another, a fog effect covers only some images. When those edited images are used to build or refine a 3D model such as a NeRF or Gaussian splat, the mismatched views produce blurry, geometrically wrong results. The paper's goal is to steer the editing process itself so that edited views stay consistent with one another, without any additional model training.

Key Contributions

1. A flexible, training-free guidance method that can steer a wide range of image editing models (multi-step diffusion, one-step diffusion, and flow-matching) toward multi-view consistent edits.
2. A correspondence-based consistency loss built on the idea that corresponding points across views should look similar after editing, optimized directly during denoising.
3. A demonstration that edited images can directly refine a Gaussian splat model, recovering fine details; the method also supports sparse (3-4 view) setups by combining with a multi-view diffusion model.
4. An experimental comparison against recent work (EditSplat and DGE) showing improved multi-view consistency of the edited images before any Gaussian splat refinement, plus edited Gaussian splat models faithful to prompts with clear detail, supported by video comparisons.

Main Findings

• Best multi-view consistency among compared methods: In the image-consistency evaluation (Table 1), the method achieves MEt3R 0.212, PSNR 23.46, SSIM 0.716 and LPIPS 0.247 with InstructPix2Pix, versus 0.243/21.19/0.679/0.271 for per-image editing, 0.329/20.20/0.641/0.389 for EditSplat, and 0.224/21.58/0.705/0.256 for DGE. Unedited images give the upper-bound reference of MEt3R 0.183, PSNR 27.36, SSIM 0.815, LPIPS 0.190.
• Works across editing backbones: With pix2pix-Turbo, per-image editing gives MEt3R 0.291 and PSNR 18.29, while the guided version reaches MEt3R 0.226 and PSNR 24.73.
• No clear gain in text fidelity, by design: On the CLIP text metrics the method is comparable to per-image editing (e.g., CLIPdir 0.152 vs 0.153, CLIPsim 0.249 vs 0.249, CLIPimage 0.817 vs 0.813 with InstructPix2Pix), which the authors say is expected because only multi-view consistency is optimized. EditSplat and DGE show higher CLIPsim/CLIPdir but lower CLIPimage, indicating stronger edits that deviate more from the input images.
• Gaussian splat editing is comparably prompt-faithful: In the 3D results (Table 2), the method's CLIPdir/CLIPsim/CLIPimage are 0.121/0.237/0.830 with InstructPix2Pix, close to per-image editing (0.126/0.239/0.833) and EditSplat (0.123/0.238/0.832). DGE reaches 0.146/0.242 but drops to 0.752 on CLIPimage and has the highest (worst) MEt3R at 0.242.
• Consistent edits sharpen Gaussian splat renderings: Inconsistently edited images produce blurrier splats; the qualitative comparisons point to sharper ears, more visible fog, and better-preserved face shape, gaze and watch when using the guided edits.
• Sparse views work: With only 3-4 input views plus ViewCrafter interpolation (Table 3), the method reaches PSNR 24.49, SSIM 0.725, LPIPS 0.281, CLIPsim 0.263 and MEt3R 0.364, versus PSNR 22.20, SSIM 0.687, LPIPS 0.290, CLIPsim 0.259 and MEt3R 0.404 for per-image editing.
• Ablations (Table 4): Using more than two previously edited images for the matching loss saturates (1 image: PSNR 22.97, MEt3R 0.394, 28 s/image; 2 images: 23.15, 0.385, 36 s; 3 images: 23.13, 0.392, 44 s). Three backward guidance steps give the best trade-off (0 steps: PSNR 21.71, 16 s; 3 steps: 23.15, 36 s; 6 steps: 23.51 but 57 s). Removing the LPIPS patch loss hurts consistency (MEt3R 0.403 at λ=0 vs 0.385 at λ=2), and raising the weight to 5 does not help (MEt3R 0.409).
• Runtime: On the IN2N "Face" scene the complete editing process takes about 22 minutes with InstructPix2Pix and about 17 minutes with pix2pix-Turbo on a single A100 GPU.

Methodology in Plain English

The method treats multi-view editing as a sequence of single-image edits that are kept in agreement with each other.

1. Edit views one at a time. Each new view is edited with an existing text-driven editor, but the denoising process is nudged using already-edited views.
2. Find corresponding points. A robust dense matcher (RoMa) finds pixels that depict the same physical point in the image being edited and in previous edits. The authors use matches between unedited images when the edit should roughly preserve geometry (textures, materials, colors, weather) and matches between edited images when the edit changes geometry (adding or replacing objects). Matches with certainty below 0.05 are discarded, up to a maximum of 50,000 matches.
3. Score disagreement with a consistency loss. Matched points are compared using an L1 term plus a perceptual (LPIPS) term on 64×64 patches centered on the matches, with the perceptual weight λ set to 2. From the current matches, 1,500 correspondences are randomly sampled for the patch loss.
4. Push the denoising toward low loss. For InstructPix2Pix, universal guidance adds the gradient of the consistency loss to the predicted noise, plus "backward guidance" that optimizes a correction to the predicted clean latent; guidance is activated only for the last N_g = 700 steps after running denoising unguided. The authors used 20 denoising steps with guidance scales s_T = 7.5 and s_I = 1.5. For the one-step model pix2pix-Turbo, a few initial seeds are optimized; for FLUX.1 (28 denoising steps), a linear approximation of the denoising trajectory avoids backpropagating through all steps.
5. Choose comparison views sensibly. When generating a new image, it is compared with the two previously edited images that have the most matching points to it.
6. Update the 3D model. Consistently edited images are used to resume training of an existing Gaussian splat model for 20 epochs (800-2500 iterations, with equal weights on L1 and LPIPS rendering losses). For the sparse case, 3-4 edited views are expanded to 50-75 additional views with the multi-view diffusion model ViewCrafter, and a Gaussian splat is trained from scratch for 5,000 iterations.

Evaluation uses 8 test scenes from EditSplat (real-world scenes from IN2N, Mip-NeRF360 and BlendedMVS) with 21 edit prompts, and 4 validation scenes (2 from Mip-NeRF360, 1 from IN2N, 1 self-captured head scene) with 7 prompts; scenes contain 65-350 images, and all experiments run on a single A100 GPU. Consistency is measured with MEt3R (which compares DINO embeddings of Dust3R matches) and by training a Gaussian splat from the edited images and computing PSNR, SSIM and LPIPS on held-out edited views; prompt fidelity is measured with CLIPdir, CLIPsim and CLIPimage.

Why This Matters

Most high-quality image editors know nothing about the 3D scene being photographed, so using them for 3D content means either expensive iterative edit-render loops or per-image results that disagree. This work offers a lightweight, model-agnostic plug-in that improves agreement between views at the denoising stage, and it does so without training, without an existing 3D representation for the image-consistency experiments, and without restricting the user to one particular editor. It also shows the approach carries through to the final 3D asset, not just to the intermediate images.

Real-world applications:

• 3D asset and scene authoring for games, film and VFX pipelines, where artists restyle a captured scene (weather, materials, time of day) and need every camera view to change the same way.
• E-commerce and product visualization, where a product photographed from many angles must be restyled consistently for a catalog or configurator.
• Digital humans and avatars, where a portrait scan is edited (hair, makeup, accessories) and the edited 3D model must still look like the same person across viewing directions.
• Sparse-capture workflows, where only a handful of photographs exist and generated intermediate views must be consistent enough to reconstruct an edited 3D scene.
• Simulation and synthetic data generation, where edited 3D scenes are needed for training perception systems with controlled appearance changes.

Industry relevance: the method slots into existing diffusion/flow editors rather than replacing them, works on a single GPU, and reports runtimes of minutes per scene, which matters for studios and pipelines already using tools such as InstructPix2Pix, pix2pix-Turbo or FLUX.1 and Gaussian splatting-based reconstruction.

Future Directions

• Stronger non-rigid and geometry-changing edits. The authors already use matches between edited images for non-rigid edits, but the main evaluation uses near-rigid edits for InstructPix2Pix and pix2pix-Turbo; extending reliable consistency to large structural changes remains open.
• Reducing guidance cost. Backward guidance with more steps gives slightly better consistency at a substantially higher per-image time (57 s/image at 6 steps vs 16 s/image at 0 steps), so cheaper or adaptive guidance schedules are a natural next step.
• Better handling of occlusion and disocclusion. Correspondences inevitably break where an edit removes or hides geometry; more robust matching or visibility reasoning could extend the framework.
• Beyond static scenes and still images. The paper edits static scenes and sparse view sets; extending the same correspondence-based loss to dynamic scenes, video editing, or joint optimization of the 3D representation and the edits is a logical direction.
• Understanding the consistency/fidelity trade-off. The paper reports no overall improvement on CLIP text metrics relative to per-image editing, so jointly optimizing consistency and prompt fidelity is an open question.

Target Audience

Researchers and graduate students working on diffusion/flow-based image editing, multi-view consistency, and 3D scene editing with NeRFs or Gaussian splatting will get the most from this paper. It is also relevant to practitioners who need to restyle captured 3D content and care about view-to-view agreement, and to anyone interested in training-free guidance techniques that adapt pretrained generative models to new objectives without retraining.

Authors’ abstract

Recent advancements in diffusion and flow models have greatly improved text-based image editing, yet methods that edit images independently often produce geometrically and photometrically inconsistent results across different views of the same scene. Such inconsistencies are particularly problematic for editing of 3D representations such as NeRFs or Gaussian splat models. We propose a training-free guidance framework that enforces multi-view consistency during the image editing process. The key idea is that corresponding points should look similar after editing. To achieve this, we introduce a consistency loss that guides the denoising process toward coherent edits. The framework is flexible and can be combined with widely varying image editing methods, supporting both dense and sparse multi-view editing setups. Experimental results show that our approach significantly improves 3D consistency compared to existing multi-view editing methods. We also show that this increased consistency enables high-quality Gaussian splat editing with sharp details and strong fidelity to user-specified text prompts. Please refer to our project page for video results: https://3d-consistent-editing.github.io/