EditInfinity: Image Editing with Binary-Quantized Generative Models

AI summary

Overview

Research area: Computer Vision — text-driven image editing, generative models, and parameter-efficient model adaptation.

Technical level: Intermediate. The paper assumes familiarity with diffusion models, image inversion, autoregressive generation, and token quantization, but its central idea can be grasped with a basic understanding of how generative models map text to images.

Scope: This paper introduces EditInfinity, a method that adapts a binary-quantized (autoregressive) generative model called Infinity for text-driven image editing, and evaluates it against diffusion-based baselines on the PIE-Bench benchmark.

What This Paper Is About

Most modern image editing systems reuse pretrained diffusion models: they first "invert" a source image back into the model's generative trajectory, then edit along that trajectory using a new text prompt. The trouble is that diffusion inversion has no exact intermediate ground truth, so approximation errors creep in and degrade editing quality.

This paper asks whether a different class of model — a binary-quantized generative model — can sidestep that problem, because such models can compute the exact discrete representation of any image directly, providing precise supervision for the inversion step. The authors build EditInfinity on top of Infinity (a binary-quantized text-to-image model) and show it outperforms diffusion baselines on PIE-Bench across add, change, and delete edits.

Key Contributions

1. Applying the inversion–editing paradigm to a binary-quantized model. EditInfinity adapts Infinity, a binary-quantized generative model, for image editing, investigating parameter-efficient adaptation of this model family rather than diffusion models.

2. An image inversion mechanism with exact supervision. The method formulates inversion as an optimization problem (a cross-entropy loss on quantized tokens) and combines text prompting rectification with image style preservation, using the exact multi-scale quantized representations of the source image as supervision.

3. A holistic smoothing strategy. A piecewise linear smoothing kernel blends source and target tokens across a multi-scale autoregressive process, aiming for high fidelity to the source image and precise semantic alignment to the target prompt.

4. Extensive evaluation on PIE-Bench. Experiments across nine editing tasks, including quantitative metrics, qualitative comparisons, a user study, runtime comparisons, and ablations.

Main Findings

• State-of-the-art editing trade-off: On PIE-Bench, EditInfinity reports PSNR 27.95, LPIPS 33.08, MSE 24.27, SSIM 92.12, whole-image CLIP 26.41, edited-region CLIP 23.47, and IR 5.88 — the best results among all compared methods in the table. The authors state the method improves the balance between background preservation and text-aligned editing.

• Best IR score: EditInfinity attains the highest IR (Image Reward) score at 5.88, which the paper describes as reflecting substantially higher editing success rates. For reference, RF-Edit scores 5.18 and Gemini scores 5.30.

• Base-model advantage is not the explanation: On the GenEval benchmark, Infinity-2B scores 0.66 overall, comparable to FLUX.1-dev (0.66) and even lower on some tasks such as two-object and counting. Despite this, the Infinity-based EditInfinity surpasses FLUX-based methods StableFlow and RF-Edit, which the authors present as evidence that the method, not just the base model, drives the gains.

• Faster iterative editing: Using a single NVIDIA L20 GPU, EditInfinity's inversion takes 107.06 seconds and each subsequent edit takes 3.64 seconds. The paper states per-edit speed is over 7x faster than other methods on average, while inversion is only 4x longer on average — front-loading cost for iterative workflows.

• User study preference: In a study with 140 images from the 'random class' of PIE-Bench, 60 volunteers each evaluated 20 editing cases, and 43.2% preferred EditInfinity's results — the highest among all approaches.

• Smoothing kernel ablation: Removing the kernel yields higher raw background metrics (PSNR 31.12, LPIPS 24.47, MSE 13.03, SSIM 93.53, whole CLIP 25.44, edited CLIP 23.12) but a much lower IR of 2.85. The linear kernel (PSNR 28.50, LPIPS 31.58, MSE 22.94, SSIM 92.36, whole 26.22, edited 23.99, IR 5.39) outperforms a Gaussian kernel (PSNR 28.15, LPIPS 32.91, MSE 24.40, SSIM 92.17, whole 26.10, edited 23.81, IR 4.61).

• Mask sensitivity: Performance is reported as not highly sensitive to mask source. With user-provided masks, EditInfinity records PSNR 28.50, LPIPS 31.58, MSE 22.94, SSIM 92.36, whole CLIP 26.22, edited CLIP 23.99, IR 5.39; with automatically generated cross-attention masks, PSNR 27.47, LPIPS 44.97, MSE 46.91, SSIM 90.30, whole CLIP 25.71, edited CLIP 23.22, IR 5.40.

• Ablation on prompt and LoRA: Removing both components causes significant structural inconsistencies. The learnable prompt improves target-prompt alignment but often shifts global style; adding LoRA restores source stylistic consistency. Training LoRA beyond 20 iterations leads to overfitting where the model ignores editing intent, so training is stopped at 20 iterations.

Methodology in Plain English

Infinity generates images in stages: an image is encoded into features, then broken into a series of quantized "residual" maps across multiple scales, using a binary spherical quantization scheme that makes each token a vector of +1/√d or -1/√d values. A transformer then predicts these maps scale by scale, conditioned on a text prompt encoded by Flan-T5.

EditInfinity reuses this machinery in two phases:

1. Inversion. Given a source image, the method computes its exact quantized tokens — something the authors highlight as directly attainable, unlike diffusion noise latents. It then optimizes a learnable prompt (20 learnable tokens plus an instruction prompt such as "the language style of this prompt is") so the model's predictions match those exact source tokens, using a cross-entropy loss over each scale. Afterwards, the prompt is frozen and LoRA — small trainable low-rank matrices added only to the FFN layers — is applied so the model captures the source image's global style with minimal overhead and without overfitting.

2. Editing. With the optimized prompt and LoRA in place, the model generates target tokens conditioned on the target prompt. Because a hard swap would cause visible splicing, the authors define a piecewise linear smoothing kernel based on Manhattan distance to the user-provided edit mask. Tokens within τ1 = 1 of the mask get weight 0 (use target content), tokens beyond τ2 = 4 get weight 1 (use source content), and tokens between are blended linearly. This blending happens at every autoregressive scale — source tokens in unedited regions guide the generation of edited regions — and the blended result feeds into the next scale. The final tokens are decoded into the edited image.

Why This Matters

Impact on research: The paper challenges the assumption that diffusion models are the natural backbone for inversion-based editing. It argues that exact, attainable quantized supervision removes a fundamental error source in the inversion step, and it opens a research direction on adapting binary-quantized and autoregressive generative models for editing tasks.

Real-world applications:

• Iterative creative workflows where a user applies many edits to the same image, since each additional edit takes only 3.64 seconds after the initial inversion.
• Photo retouching and object removal/addition, using the add, change, and delete operations evaluated on PIE-Bench.
• Content production pipelines where maintaining background fidelity while changing a specified region matters, e.g., product or advertising imagery.
• Automated or semi-automated editing with automatically generated masks via cross-attention, useful when users cannot draw masks by hand.

Industry relevance: The method is parameter-efficient (LoRA plus a learnable prompt), and its runtime profile — expensive inversion, cheap repeated editing — suits interactive tools and batch workflows where many variations of one source image are requested.

Future Directions

• Style change and dense structural patterns. The authors' own limitation section notes that when no background needs preserving and the image has detailed structural patterns, blending between source and target tokens is constrained and structural fidelity may degrade. Addressing this blending limitation is a clear next step.

• Beyond masks. The paper defaults to user-provided masks but demonstrates automatic cross-attention masks. Reducing the remaining gap between the two (LPIPS 44.97 for automatic versus 31.58 for user masks) is an open problem.

• Extension to other binary-quantized or autoregressive models. EditInfinity is built specifically on Infinity-2B; whether the inversion-with-exact-supervision recipe transfers to other quantized generators is untested here.

• LoRA training schedule. The paper reports that training LoRA beyond 20 iterations causes overfitting and loss of edit responsiveness; a more principled way to balance editability against fidelity is not established.

Target Audience

This paper suits researchers and graduate students in generative computer vision working on image editing, inversion methods, or autoregressive image generation, as well as practitioners who build editing tools and care about the speed-versus-fidelity trade-off in iterative editing. Readers already comfortable with diffusion inversion, latent tokenization, and LoRA will get the most out of it; those new to the field should first understand the standard inversion–editing pipeline that this work departs from.

Authors’ abstract

Adapting pretrained diffusion-based generative models for text-driven image editing with negligible tuning overhead has demonstrated remarkable potential. A classical adaptation paradigm, as followed by these methods, first infers the generative trajectory inversely for a given source image by image inversion, then performs image editing along the inferred trajectory guided by the target text prompts. However, the performance of image editing is heavily limited by the approximation errors introduced during image inversion by diffusion models, which arise from the absence of exact supervision in the intermediate generative steps. To circumvent this issue, we investigate the parameter-efficient adaptation of binary-quantized generative models for image editing, and leverage their inherent characteristic that the exact intermediate quantized representations of a source image are attainable, enabling more effective supervision for precise image inversion. Specifically, we propose EditInfinity, which adapts \emph{Infinity}, a binary-quantized generative model, for image editing. We propose an efficient yet effective image inversion mechanism that integrates text prompting rectification and image style preservation, enabling precise image inversion. Furthermore, we devise a holistic smoothing strategy which allows our EditInfinity to perform image editing with high fidelity to source images and precise semantic alignment to the text prompts. Extensive experiments on the PIE-Bench benchmark across `add', `change', and `delete' editing operations, demonstrate the superior performance of our model compared to state-of-the-art diffusion-based baselines. Code available at: https://github.com/yx-chen-ust/EditInfinity.