Low-Resolution Editing is All You Need for High-Resolution Editing

AI summary

Overview

Research area: Computer vision and generative modeling, specifically controllable image editing with diffusion and transformer-based generative models at resolutions beyond their native operating scale.

Technical level: Advanced. The paper assumes familiarity with diffusion models, DDIM sampling, Null-text inversion, latent space representations, and patch-based inference.

Scope: The paper proposes ScaleEdit, a test-time optimization framework that performs high-resolution image editing by editing at low resolution with an existing editor and then transferring fine-grained source details patch-by-patch through a learnable function inside a pretrained generative model.

What This Paper Is About

Existing image editing methods based on diffusion and transformer generative models operate at fixed low resolutions, typically supporting only up to 1K (approximately 1024²). The paper introduces the task of high-resolution image editing: given a high-resolution source image and a low-resolution edited reference produced by an off-the-shelf editor, generate a high-resolution result that carries the reference semantics while preserving the source image's fine-grained texture. The naive alternative, editing at low resolution and then applying super-resolution methods, is argued to fail because the source details are never conditioned during the super-resolution stage.

Key Contributions

1. A new task definition. The authors propose high-resolution image editing as a task and support it with a patch-wise inference mechanism, which they state is the first work to do so.
2. A detail transfer module. They introduce a feature modification term defined as a learnable 1×1 convolution applied to intermediate features of a pretrained generative model, optimized at test time to steer a low-resolution trajectory toward a high-resolution source trajectory.
3. A synchronization strategy. They design a synchronization method operating across non-overlapping patches that combines blended Tweedie updates with a resampling strategy, avoiding the need for overlapping inference and thereby reducing computational cost.
4. Empirical validation. They report state-of-the-art editing capacity against four diffusion-based super-resolution baselines under 1K- and 2K-editing scenarios, plus qualitative results at 8K and generalization to a second backbone.

Main Findings

• ScaleEdit leads the reported quantitative comparison. With the pretrained Stable Diffusion v2.1-base backbone, ScaleEdit attains HaarPSI 0.342, M-MSE 0.054, M-SSIM 0.739, M-PSNR 22.132, and LPIPS 0.460 in the 1K-editing scenario, compared with DiT-SR (0.335, 0.058, 0.695, 21.528, 0.477), DiT4SR (0.324, 0.060, 0.625, 20.740, 0.509), PiSA-SR (0.328, 0.058, 0.668, 21.273, 0.465), and TSD-SR (0.329, 0.061, 0.649, 20.766, 0.489).
• The same ordering holds in the 2K-editing scenario for four of five metrics. ScaleEdit reports HaarPSI 0.331, M-MSE 0.053, M-SSIM 0.806, M-PSNR 21.955, and LPIPS 0.496; the baselines are DiT-SR (0.316, 0.057, 0.754, 21.380, 0.507), DiT4SR (0.305, 0.058, 0.684, 20.701, 0.534), PiSA-SR (0.312, 0.056, 0.755, 21.320, 0.472), and TSD-SR (0.312, 0.059, 0.715, 20.796, 0.514). In that 2K setting PiSA-SR records the lowest LPIPS at 0.472 versus 0.496 for ScaleEdit.
• Synchronization is necessary to avoid visible seams. The ablation study reports that without synchronization, results show boundary artifacts, visible seams, and inconsistencies along patch borders, while synchronization enforces coherent denoising trajectories and artifact-free transitions.
• A global constant offset is insufficient; the transfer function must adapt to the feature. The authors report that parameterizing the transfer function as a learnable constant vector fails to produce reliable transformations, particularly when source and reference differ semantically, giving cat-to-dog as an example, which motivated the 1×1 convolution formulation.
• The transfer function is applied only up to a controllable timestep. A hyperparameter τ governs how far into the reverse process detail injection occurs, with the modification set to zero for all timesteps beyond τ; all reported experiments use τ = 15 and a total of T = 50 timesteps.
• The method generalizes across architectures and resolutions. It is demonstrated on Stable Diffusion v2.1-base and on FLUX.1-dev, which uses a transformer-based architecture, and qualitative results are shown for an 8K-editing scenario without additional tuning.
• Masked metrics indicate more source-aware synthesis. The masked variants M-MSE, M-SSIM, and M-PSNR, computed with masks generated by LANG-SAM using background masks for object-change tasks and foreground masks for background-modification tasks, favor ScaleEdit, which the authors attribute to super-resolution pipelines being source-unconditional.

Methodology in Plain English

The pipeline assumes three inputs: a high-resolution source image, a downsampled low-resolution version of it, and a low-resolution reference produced by running a standard low-resolution editor, such as Nano Banana, on the downsampled source.

First, all three images are resized to the high-resolution source's dimensions and cut into non-overlapping N×M patches whose size matches the native input resolution of the pretrained generative model, so N = H_h/H_d and M = W_h/W_d. Each patch is encoded with the pretrained VAE encoder, and forward diffusion trajectories are computed for the high-resolution source patch, the low-resolution source patch, and the low-resolution reference patch, optionally enhanced with Null-text inversion for accurate reconstruction.

Second, the method learns a transfer function that nudges the low-resolution trajectory to follow the high-resolution source trajectory. This function modifies an intermediate feature inside the network, either the output of a ResNet block on the upward path of a U-Net or the output of the final linear layer after a single-stream block in a transformer. It is implemented as a 1×1 convolution over that feature, which mixes information across channels while keeping the spatial layout intact, and it is optimized per patch and per timestep by minimizing the squared error between the reversed latent and the corresponding high-resolution source latent. The modification is applied only for timesteps up to τ, which acts as a dial trading detail transfer against content preservation.

Third, because patches are denoised independently, their trajectories can diverge and produce seams. The synchronization step builds an auxiliary latent by spatially blending the bottom half of one patch with the top half of its neighbor, computes Tweedie estimates for the auxiliary latent and the original patches, and blends them with a linearly increasing interpolation weight so the transition across the border is smooth. To avoid a costly extra optimization for the auxiliary latent, a resampling strategy re-noises the detail-injected latents with a forward step that does not re-inject the transfer function, decoupling synchronization from detail injection.

Finally, the N×M resulting latents are spatially merged and decoded with the pretrained VAE decoder to produce the high-resolution edited image.

Why This Matters

The work pushes controllable image editing past the 1K ceiling that current methods are confined to, and it argues that the standard two-stage recipe of low-resolution edit plus super-resolution cannot recover micro-scale texture because source details are absent from the super-resolution conditioning. Framing this as a task, rather than a single model, gives the community a target to measure against.

Real-world applications include:

• Professional and consumer photo editing, where users want to modify large, high-resolution photographs while keeping skin, fabric, and material textures intact.
• Design and marketing content creation, where brand assets and product imagery are produced and retouched at print or display resolutions above 1K.
• Document processing, listed among the domains where high-resolution visual content is essential.
• Autonomous driving and 3D shape generation, also cited as high-resolution visual workflows, where faithful structural preservation matters alongside instruction-following edits.

Industry relevance: the approach is a test-time framework with no model training, and it is demonstrated on top of two widely used backbones (Stable Diffusion v2.1-base and FLUX.1-dev), so it can be layered onto existing editing stacks. The design choice to synchronize non-overlapping patches removes the overlapping-inference cost that prior synchronization schemes incur, which matters for serving high-resolution edits at practical latency.

Future Directions

• Extend the framework to other modalities and settings. The paper situates itself among high-resolution image synthesis, video generation, and 3D shape generation, leaving high-resolution video and 3D editing as open extensions of the patch-wise detail transfer idea.
• Reduce the cost of test-time optimization. The method optimizes a transfer function per patch and per timestep up to τ, so improving optimization efficiency or amortizing it across images is a natural next step.
• Broaden evaluation. The reported benchmark uses 100 source images generated with FreeScale at 4K, downsampled to 2K, 1K, and 512², with 400 conditioning-image and text-instruction pairs per scenario; no user study or runtime measurement is reported, and a larger and more diverse benchmark would test the generality of the results.
• Address residual perceptual gaps. ScaleEdit does not record the lowest LPIPS in the 2K scenario, so understanding and closing perceptual-similarity differences relative to baselines such as PiSA-SR remains an open question.

Target Audience

Researchers working on diffusion-based and transformer-based image editing, image-to-image translation, and super-resolution will find the task formulation and the patch synchronization mechanism most useful. Practitioners building high-resolution image editing or content creation products, and engineers evaluating whether to add a super-resolution stage after a low-resolution editor, also benefit from the quantitative comparison against DiT-SR, DiT4SR, PiSA-SR, and TSD-SR. Readers without background in diffusion sampling will need to consult the preliminary section on DDIM and Null-text inversion to follow the methodology.

Authors’ abstract

High-resolution content creation is rapidly emerging as a central challenge in both the vision and graphics communities. Images serve as the most fundamental modality for visual expression, and content generation that aligns with the user intent requires effective, controllable high-resolution image manipulation mechanisms. However, existing approaches remain limited to low-resolution settings, typically supporting only up to 1K resolution. In this work, we introduce the task of high-resolution image editing and propose a test-time optimization framework to address it. Our method performs patch-wise optimization on high-resolution source images, followed by a fine-grained detail transfer module and a novel synchronization strategy to maintain consistency across patches. Extensive experiments show that our method produces high-quality edits, facilitating high-resolution content creation.