SONIC: Spectral Optimization of Noise for Inpainting with Consistency

AI summary

Overview

Research area: Computer vision — generative models and inverse problems, specifically image inpainting with pretrained text-to-image diffusion/flow models.

Technical level: Intermediate. The paper assumes familiarity with denoising diffusion and flow models, classifier-free guidance, latent spaces, and metrics such as FID and LPIPS.

Scope (one sentence): SONIC is a training-free inpainting method that optimizes only the initial noise sample of an off-the-shelf flow model — using a linearized denoising trajectory and spectral-domain Adam optimization — so that the unmasked regions of an image are reproduced before a conventional inpainting pipeline is run.

What This Paper Is About

Guidance-based training-free methods theoretically let any generic text-to-image model solve inverse problems such as inpainting, but in practice they underperform specialized inpainting models, which is why dedicated inpainting models are usually trained. The authors argue that the missing ingredient is proper optimization of the initial noise sample: the seed noise largely determines the structure of the final generated image, so a seed that already fits the unmasked content produces far more consistent inpainting. SONIC finds such a seed in as few as tens of optimization steps, then hands it to a conventional training-free inpainter.

Key Contributions

1. Linearization of the denoising trajectory. The authors approximate the entire T-step denoising (flow) path with a single linear equation using a stop-gradient operator (Eq. 2), which removes the need to back-propagate through the denoiser. The paper notes that with an NVidia RTX 5090, back-propagating through more than one denoising step for StableDiffusion 3.5 is not feasible — this cost is likely why the initial-noise relationship was previously overlooked.

2. Spectral preconditioning. Instead of treating the initial noise x_T as the trainable parameter, they optimize its Fourier transform X_T with the Adam optimizer, so each frequency is scaled by its own second moment and can converge at its preferred pace. The resulting objective is given in Eq. 4.

3. Spatial constraining of gradient updates. Gradient updates are restricted to observed (unmasked) pixels, because a single frequency component affects all pixels and unobserved latents can otherwise diverge away from the Gaussian distribution the denoising model expects, causing visible artifacts and mask borders.

4. Latent-space operation with nearest-neighbor fill. The method works entirely in the latent space and simply fills masked region pixels with their nearest-neighbor color before encoding, avoiding an expensive VAE bottleneck; an ablation shows this matches ground-truth-pixel filling as an upper bound.

Main Findings

• State-of-the-art on all three benchmarks for perceptual metrics. On FFHQ, DIV2K, and BrushBench, the method outperforms all baselines in SSIM, LPIPS, and FID. On FFHQ it scores SSIM 0.857, LPIPS 0.121, FID 12.883, versus BrushNet at 0.759 / 0.237 / 17.950 and FLAIR at 0.823 / 0.266 / 18.982. On DIV2K it scores 0.803 / 0.115 / 17.699; on BrushBench 0.861 / 0.153 / 48.072.

• Best FID by a large margin on FFHQ. FID 12.883, with BrushNet second best (17.950) and FLAIR third (18.982). The authors attribute BrushNet's favorable FID partly to its ability to hide seams, while noting it performs significantly worse on SSIM and LPIPS, where structure is considered.

• Competitive human-preference metrics. On FFHQ the method is best in IR (0.280) and HPS v2 (0.245) and second best in AS (5.848). On BrushBench it is third best on the non-perceptual metrics, where BrushNet leads in IR (1.246), HPS v2 (0.271), and AS (6.366) — expected, since BrushBench was developed by BrushNet.

• PSNR is not a reliable signal here. The paper explicitly warns that lower PSNR does not mean worse inpainting, since many valid answers exist; FLAIR records high PSNR on DIV2K (23.128) but, as shown qualitatively, produces blurry patches. FlowDPS obtains the best CLIP on FFHQ (25.076) while having far worse LPIPS (0.374) and FID (48.333).

• Human study favors the method. In a single-blind two-alternative choice study with N=37 raters, SONIC was preferred in 90% of comparisons.

• Spectral preconditioning is the decisive design choice. Without it (optimizing in the spatial domain with a learning rate of 0.05), FFHQ LPIPS degrades from 0.121 to 0.127 and FID from 12.883 to 14.579, with image flickering between optimization steps as spatial frequencies converge at different paces.

• Spatial constraining is also critical. Removing it collapses performance: PSNR 22.562 to 18.863, SSIM 0.857 to 0.833, LPIPS 0.121 to 0.165, FID 12.883 to 33.269, and IR from 0.280 to -0.259.

• Nearest-neighbor fill is sufficient. Optimizing with the ground-truth image fed to the encoder gives SSIM 0.856, LPIPS 0.121, FID 12.773, IR 0.280 — essentially identical to the full method's 0.857 / 0.121 / 12.883 / 0.280.

• Convergence behavior. Using the first 100 DIV2K prompts with StableDiffusion 3.5 at CFG 2.0 and T=20, starting from the same random point drawn from N(0,1), Adam in the spectral domain converges well regardless of learning rate; Adam in the spatial domain converges much more slowly and becomes unstable at large learning rates; SGD performs identically in both domains because the Fourier transform is linear.

• Runtime comparable to a low-NFE baseline. Approximately one minute to inpaint one image on a GeForce RTX 5090, roughly the same time as FLAIR using 50 NFE, despite the 400 NFE used by the method.

• Baselines fail in characteristic ways. BLD-SD3.5 fails to align global structure; BrushNet hides seams but misaligns content, and on DIV2K produces caption-like results in horizontal stripe masks, hinting its outputs correlate with mask shape; FLAIR often cannot match global structure and produces disjoint or blurry inpainting.

Methodology in Plain English

The starting point is a simple least-squares question: which initial noise, when denoised and then corrupted by the mask, best matches the observed image (Eq. 1)? Answering it directly would mean unrolling the whole denoising chain and back-propagating through every step, which is impractical for large modern models. The authors instead replace the denoising path with a straight line between the initial noise and its denoised output, treating the difference between the two as a constant (Eq. 2). This makes the objective differentiable with respect to the initial noise without ever touching the denoiser — the resulting loss (Eq. 3) is effectively a straight-through estimator that drops Jacobians.

Applying that loss naively is unstable, so the method optimizes the noise in the spectral domain with Adam rather than in pixel/latent space. Because Adam divides each parameter by its second moment, and each parameter is now a frequency, different spatial frequencies effectively receive different step sizes — a preconditioning that lets each converge at its own natural rate. To keep the optimized noise on the manifold of plausible seeds, gradient updates are masked so only observed pixels move. The masked image is encoded by filling the masked region with nearest-neighbor colors, and the whole procedure runs on StableDiffusion 3.5 with BLD-SD3.5 as the downstream inpainter, Adam at a learning rate of 3.0 with beta1 = 0.9 and beta2 = 0.999, classifier-free guidance scale 2.0, resolution 1024x1024, and T = 20 denoising steps.

Why This Matters

The paper suggests that a widely assumed limitation of training-free methods — that generic models simply cannot match specialized inpainting models — is instead an optimization oversight. If correct, that reframing is valuable because it avoids the cost of training or fine-tuning task-specific models and avoids their tendency to fail on mask types unseen during training.

Real-world applications:

• Photo editing and restoration: removing objects or people from personal or archival photographs while preserving lighting and layout.
• Creative production: filling masked regions in design, advertising, and film workflows where global structure and lighting consistency matter more than exact pixel match.
• Medical and scientific imaging: recovering missing regions in a scan where the surrounding anatomy must constrain the reconstruction.
• Video post-production: the authors report preliminary video inpainting results in the supplementary material, targeting object removal across frames.

Industry relevance: the method only manipulates the initial noise sample and leaves the denoising process untouched, so it drops into existing pipelines that already use off-the-shelf models such as StableDiffusion 3.5 and BLD-SD3.5. Its runtime — roughly a minute per image on a GeForce RTX 5090 — is comparable to FLAIR at 50 NFE, which keeps it within reach of deployed editing tools. The industrial backing of the work (Google DeepMind co-authorship, plus Google, NSERC, and the Digital Research Alliance of Canada in the acknowledgements) signals practical interest in reducing reliance on specialized inpainting checkpoints.

Future Directions

• Extending beyond inpainting. The authors state the method was demonstrated purely on inpainting but believe the linearization and spectral preconditioning could in principle apply to super-resolution, deblurring, and video inpainting; preliminary video inpainting results are shown in the supplementary material. They hope the work becomes a starting point for using generic flow models on other inverse problems.
• Characterizing when optimization alone is enough. Since seeded noise may not exactly satisfy the masking equation, open questions remain about how far an enhanced starting point can carry a fixed downstream denoiser.
• Better understanding of the linearity assumption. The approach treats D_T(x_T) - x_T as constant over the trajectory, justified by flow models generally being trained to produce linear trajectories; the paper points to an illustrative example in the supplementary material but does not quantify the approximation error for other model families.
• Reducing optimization cost further. The paper reports diminishing gains as optimization iterations increase and includes an ablation on the number of optimization steps in the supplementary material, leaving room to determine the minimal iteration budget — the teaser figure shows results at 0, 5, 10, 20, and 19 iterations, but the number of optimization iterations used for the main quantitative results is not explicitly stated in the main text.

Target Audience

Researchers and graduate students working on diffusion and flow models, inverse problems, and training-free guidance methods will find the core methodological arguments most useful. Practitioners building image-editing or object-removal products who want to avoid training or fine-tuning inpainting-specific models will benefit from the practical framing and runtime figures. Readers already familiar with posterior sampling methods such as FLAIR, FlowChef, and FlowDPS, or with trained approaches such as BrushNet, will be best positioned to judge the empirical claims against the reported baselines.

Authors’ abstract

We propose a novel training-free method for inpainting with off-the-shelf text-to-image models. While guidance-based methods in theory allow generic models to be used for inverse problems such as inpainting -- in practice their effectiveness is limited, leading to the necessity of specialized inpainting-specific models. In this work, we argue the missing ingredient for training-free generic model usage is proper optimization of the initial noise sample. We optimize the initial noise to approximately reproduce the unmasked image, in as few as tens of optimization steps, then use it with a conventional training-free inpainting method. Critically, we propose two core ideas that make this possible: (i) we perform linear approximation that avoids the costly and often impractical unrolling required to relate the initial noise sample to model output -- which potentially is why this relationship was previously overlooked; and (ii) perform spectral preconditioning by optimizing the initial noise sample in the spectral domain with Adam, which stabilizes the optimization. We demonstrate our method on various inpainting tasks, outperforming the state of the art. Project website: https://ubc-vision.github.io/sonic/