Efficient Zero-Shot Inpainting with Decoupled Diffusion Guidance

AI summary

Overview

• Research area: Computer vision / generative modeling — specifically zero-shot image inpainting (and local image editing) using pretrained diffusion models as priors.
• Technical level: Advanced. The paper is written in the language of Bayesian inverse problems, diffusion/DDIM transition kernels, twisting functions, and likelihood surrogates. The core idea is intuitive, but the derivations and notation assume familiarity with diffusion model theory.
• Scope: The paper proposes a training-free ("zero-shot") inpainting method called DInG (Decoupled INpainting Guidance) that removes the expensive backpropagation through the denoiser network required by prior guidance schemes, and evaluates it on three datasets with Stable Diffusion 3.5 (medium).

What This Paper Is About

Zero-shot inpainting uses a pretrained (frozen) diffusion model to fill in missing pixels of an image, enforcing consistency with the pixels that are already visible — without retraining the model for the task. The leading zero-shot methods approximate the ideal guidance score using a surrogate likelihood evaluated at the denoiser's own output, which forces a vector–Jacobian product (VJP) through the denoiser network at every reverse diffusion step, inflating memory use and runtime. This paper's goal is to keep the reconstruction quality and observation fidelity of those methods while eliminating the backpropagation entirely, producing a faster and lighter sampler.

Key Contributions

1. A VJP-free zero-shot inpainting framework (DInG). The authors replace the standard likelihood surrogate with an alternative one in which the noise predictor is evaluated at an independent draw from the pretrained reverse transition, rather than at the current state. This "decoupling" breaks the dependency that forced gradients through the denoiser.
2. Closed-form, exactly samplable posterior transitions. The resulting surrogate has a mixture structure, and for the Gaussian inpainting likelihood the mixture components are Gaussian in closed form (Equation 3.4). Samples can therefore be drawn exactly, with the masked and unmasked coordinates sampled conditionally independently — no variational inference or sequential Monte Carlo machinery is needed.
3. A practical latent-space implementation. Because large diffusion models operate in a compressed latent space, the authors build the latent mask by average pooling the pixel-space mask (kernel and stride equal to the encoder's downsampling factor, given as s = 8 in the cited latent diffusion work), assigning each latent site the fraction of unmasked pixels in its receptive field and thresholding (typically 0.5). The method uses only one hyperparameter, the DDIM standard-deviation schedule; all experiments use η_t = σ_t(1 − α_t).
4. Extensive empirical comparison. DInG is benchmarked against a long list of guidance baselines on three datasets at low NFE budgets, reported to achieve the best fidelity/realism trade-off, lower runtime and memory, and to outperform an SD 3 model fine-tuned for image editing despite requiring no task-specific training. Code is released at the linked GitHub repository.

Main Findings

• Guidance without backpropagation: The decoupled surrogate (evaluated at an independent draw Z_s from the pretrained transition, Equation 3.3) yields posterior transitions that can be sampled exactly as a product of two Gaussians — one over the masked coordinates, one over the observed coordinates (Equation 3.4). This is the mechanism that removes vector–Jacobian products and denoiser backpropagation.
• Lowest measured inference cost in the comparison table: On DIV2K at 768px on H100 GPUs, DInG is listed at 2.9 s and 22.09 GB (Table 1), the smallest runtime among the listed methods and equal to the smallest reported memory. For reference, the other listed entries include Blended-Diff (3.0 s, 22.09 GB), DAPS (9.1 s, 22.09 GB), DDNM (3.1 s, 22.09 GB), DiffPIR (3.1 s, 22.09 GB), FlowChef (3.0 s, 22.09 GB), FlowDPS (3.0 s, 22.10 GB), PnP-Flow (3.1 s, 22.09 GB), PSLD (7.4 s, 24.49 GB), RedDiff (3.1 s, 22.09 GB), and ReSample (8.1 s, 24.50 GB).
• Best reported fidelity/realism trade-off across three benchmarks: The paper states that across FFHQ, DIV2K, and PIE-Bench, DInG achieves the best trade-off between fidelity to the visible content and realism of the reconstructions, and consistently outperforms state-of-the-art guidance methods under low NFE budgets.
• Strong zero-shot performance against a fine-tuned model: The authors report that DInG, with no task-specific fine-tuning, outperforms an SD 3 model that has been fine-tuned for image editing.
• Fixed low-NFE evaluation budget: All methods are evaluated at 50 NFEs. Because DInG requires two denoiser evaluations per diffusion step, it is run for 25 steps to match that budget. The authors note that baselines requiring VJPs are reported with their actual runtime and memory to reflect effective inference cost rather than nominal NFE counts.
• Setup and data scale: Experiments use Stable Diffusion 3.5 (medium) with CFG scale set to 2. FFHQ uses the first 5k images with the prompt "a high-quality photo of a face"; DIV2K uses its training and validation splits (900 images total) with captions generated by BLIP-2; PIE-Bench starts at 700 images of resolution 512×512 and, after removing cases where the mask completely covers the image, ends with 556 images. FFHQ and DIV2K images are resized to 768×768. PSNR-style consistency is measured as context PSNR (cPSNR) over the unmasked region only, with σ_y = 0.01 set across all tasks.
• Metrics used: FID and patch FID (pFID; 10 random 256×256 patches per image, giving 50k patches total), cPSNR, LPIPS against ground truth, and for PIE-Bench also CLIP-Score on the full image (CLIP) and on the edited region (CLIP-ED). The authors caution that no single metric should be interpreted alone.
• Partial quantitative results in the supplied text: Table 2 begins with FFHQ at 768×768 with 5k samples, showing per-mask results for several baselines (for example, on the Half mask, FID ranges from 12.1 for DiffPIR to 116.3 for PSLD among the rows shown). The DInG row of Table 2 and the entire DIV2K result block are beyond the truncation and are not reported in the content available here.
• Visual claims: Figure 1 shows DInG edits at 50 NFEs with Stable Diffusion 3.5 (medium) across different masking patterns and prompts; Figure 2 shows FFHQ and DIV2K reconstructions at 50 NFEs; Figure 3 illustrates that masking in the latent space translates to masking in pixel space despite the decoder's nonlinearity.
• Note on baselines: The text states that comparisons are against "seven state-of-the-art baselines" but then names ten methods: FlowChef, FlowDPS, DAPS, RedDiff, ReSample, PSLD, PnP-Flow, DiffPIR, DDNM, and Blended-Diff.

Methodology in Plain English

Inpainting is framed as a Bayesian inverse problem: the pretrained diffusion model supplies a prior over natural images, and a Gaussian likelihood on the observed (unmasked) pixels enforces fidelity to the reference image. Sampling from the resulting posterior requires knowing how the likelihood behaves at every noise level, which is intractable in closed form.

Prior zero-shot methods approximate that likelihood by plugging the denoiser's current best guess of the clean image into the observation likelihood. Because the guess depends on the current noisy state, differentiating this approximation requires backpropagating through the denoiser at every step — the source of the cost.

The authors' trick is to change where the denoiser is evaluated. Instead of evaluating the noise predictor at the current state, they first draw an independent sample Z_s from the ordinary pretrained reverse transition and evaluate the noise predictor at Z_s. Now the likelihood surrogate depends on the current state only through simple algebraic terms, so the resulting reverse transition becomes a mixture whose components are Gaussian. Sampling it is done in two elementary steps: draw Z_s from the pretrained transition, then sample the masked coordinates and the observed coordinates separately from the two resulting Gaussians, where the observed coordinates are a convex combination (weighted by γ) of the ordinary DDIM mean and the rescaled observation, plus appropriately scaled noise.

The method uses one hyperparameter — the DDIM standard-deviation schedule η_t — which controls stochasticity and matters most when only a handful of function evaluations are available. To run on high-resolution latent diffusion models, the pixel-space binary mask is average-pooled down to the latent resolution, with each latent site marked observed when the majority of its underlying pixels are unmasked.

Why This Matters

The paper targets the central practical tension in zero-shot editing: methods that avoid retraining usually pay for it at inference time, sometimes being slower than simply training a dedicated conditional model. By making the zero-shot route cheap, this work makes plug-and-play diffusion priors viable in latency- and memory-constrained settings, and it reframes an approximation choice (where to evaluate the denoiser) as a lever for computational efficiency rather than a fixed design.

Potential real-world applications:

• Photo restoration and repair — filling in damaged, occluded, or missing regions of historical and personal photographs while preserving surrounding content.
• Interactive image editing and content creation — allowing users to paint a mask and describe the desired content, as illustrated in the paper's example mask-and-prompt edits.
• Object removal and scene completion — deleting unwanted objects and generating coherent replacements in design and e-commerce pipelines.
• High-resolution scientific and medical imaging restoration — the paper notes the plug-and-play paradigm has been investigated from image restoration to scientific imaging, where retraining per task is often impractical.

Industry relevance: the reported 2.9 s runtime and 22.09 GB memory footprint on 768px images on an H100 (Table 1) place the method in a range where interactive or batch production inference becomes plausible, and removing the per-step backpropagation through the denoiser reduces both the cost of serving diffusion models and the engineering effort needed to add new editing tasks. The method's single hyperparameter lowers the tuning burden relative to baselines that require per-dataset hyperparameter tuning, as the authors explicitly perform for their comparisons.

Future Directions

• Theory for the decoupled surrogate. The paper takes the approximation as given and derives its transitions; a follow-up work by Ghorbel et al. (2026) is cited as providing further insight into the approximation. A tighter characterization of when the surrogate is accurate (for example, its behavior at small σ_y or few steps) remains open.
• Extension beyond inpainting. Only masked-region completion is addressed here; whether the same decoupling applies to other inverse problems such as deblurring, super-resolution, or scientific imaging — regimes the paper itself mentions as plug-and-play territory — is not established in the provided content.
• Better and more robust η schedules. The single hyperparameter η_t was fixed to σ_t(1 − α_t) in all experiments; the paper reports an ablation of this parameter in Section 4 but does not report a learned or adaptive scheme. An adaptive rule tied to the mask or noise level is a natural next step.
• Scaling and evaluation. Results here cover Stable Diffusion 3.5 (medium), CFG scale 2, three datasets, and rectangular masks for FFHQ/DIV2K with irregular masks only on PIE-Bench. Behavior under other backbones, other resolutions, and video or 3D editing setups is not reported.

Target Audience

Researchers and practitioners in generative modeling and computer vision who work on diffusion-based inverse problems, image editing, or efficient inference. It is most useful to readers already comfortable with diffusion sampling and Bayesian posterior sampling, but the central insight — evaluate the denoiser at an independent draw to avoid backpropagation — is compact enough to be of interest to engineers implementing inpainting pipelines who need faster, lighter inference without fine-tuning a model.

Authors’ abstract

Diffusion models have emerged as powerful priors for image editing tasks such as inpainting and local modification, where the objective is to generate realistic content that remains consistent with observed regions. In particular, zero-shot approaches that leverage a pretrained diffusion model, without any retraining, have been shown to achieve highly effective reconstructions. However, state-of-the-art zero-shot methods typically rely on a sequence of surrogate likelihood functions, whose scores are used as proxies for the ideal score. This procedure however requires vector-Jacobian products through the denoiser at every reverse step, introducing significant memory and runtime overhead. To address this issue, we propose a new likelihood surrogate that yields simple and efficient to sample Gaussian posterior transitions, sidestepping the backpropagation through the denoiser network. Our extensive experiments show that our method achieves strong observation consistency compared with fine-tuned baselines and produces coherent, high-quality reconstructions, all while significantly reducing inference cost. Code is available at https://github.com/YazidJanati/ding.