FIA-Edit: Frequency-Interactive Attention for Efficient and High-Fidelity Inversion-Free Text-Guided Image Editing

AI summary

Overview

Research area: Text-guided image editing with diffusion and flow-based generative models, plus a clinical extension to surgical data augmentation.

Technical level: Advanced. The paper assumes familiarity with Rectified Flow, latent diffusion, Diffusion Transformers, attention layers, classifier-free guidance, and frequency-domain filtering.

Scope: The paper proposes an inversion-free editing framework that adds explicit frequency-domain and attention-level interaction between source and target branches, evaluates it on PIE-Bench against 12 baselines, and applies it to bleeding-image augmentation for surgical classification.

What This Paper Is About

Inversion-free image editing methods are fast because they skip the step of converting a source image back into noise, but the paper argues they integrate source information poorly, causing weak background preservation, spatial inconsistency, and over-editing. The authors build FIA-Edit on the FlowEdit inversion-free backbone so that source features are explicitly injected into the target velocity field computation, both in the frequency domain and inside attention layers. The goal is high-fidelity, semantically precise edits at low computational cost, and the paper further tests whether such editing can serve as medical data augmentation.

Key Contributions

1. FIA-Edit framework: An efficient inversion-free text-guided image editing framework that explicitly preserves background structure rather than relying only on the implicit relationship between source and target velocity fields.

2. Frequency-Interactive Attention: A unified mechanism with two components — Frequency Representation Interaction (FRI), which exchanges frequency components between source and target features inside self-attention, and Feature Injection (FIJ), which injects source queries, keys, values, and text embeddings into the target branch's cross-attention.

3. Benchmark evaluation: Extensive experiments on PIE-Bench reporting state-of-the-art results across diverse editing tasks, with the best average rank of 1.7 across all reported metrics.

4. First clinical extension: The authors state they are the first to apply general-purpose text-guided image editing to clinical images, synthesizing anatomically coherent hemorrhage variations in surgical images and improving downstream bleeding classification.

Main Findings

• Editing quality and background preservation (PIE-Bench, Table 1): FIA-Edit (SD3.5 backbone) achieves Structure Distance 10.34 ×10³, PSNR 27.32, LPIPS 55.02 ×10³, MSE 28.66 ×10⁴, SSIM 89.21 ×10², CLIP similarity 25.89 (whole image) and 22.82 (edited region), giving an average rank of 1.7 — the best overall. FlowEdit, the inversion-free baseline it builds on, scores Structure Distance 23.62, PSNR 23.21, LPIPS 93.81 ×10³, MSE 69.95 ×10⁴, SSIM 85.09 ×10², CLIP 26.78 (whole) and 23.73 (edited), with rank 6.1.

• Best-in-class per-metric positions: FIA-Edit is best on Structure Distance, PSNR, and MSE; second-best on LPIPS and SSIM; second-best on whole-image CLIP; third-best on edited-region CLIP. DNAEdit ranks second overall (3.1) and P2P ranks worst (5.3).

• Trade-off in baselines: P2P has relatively good background consistency (Structure Distance 11.65 ×10³, best among non-FIA methods) but weak prompt alignment (CLIP 25.02 whole, 22.10 edited), which the authors attribute to over-retention of source content.

• Runtime and memory (Table 2): On a single RTX 4090, FIA-Edit runs at 6.30 s per edit with 17.93 GB memory, compared to FlowEdit's 3.49 s at 17.93 GB. RF-Inv (69.22 GB, 76.74 s), StableFlow (35.39 GB, 26.07 s), and RF-Edit (32.91 GB, 34.51 s) were run on an A100 80GB due to high memory demand. LDM baselines ranged from FreeDiff at 6.08 GB / 17.41 s to FlexiEdit at 18.73 GB / 38.97 s. The abstract reports approximately 6 s per 512×512 image on an RTX 4090.

• Ablation (Table 3): Adding FIJ alone to the backbone improves Structure Distance from 23.62 to 14.89, PSNR from 23.21 to 25.59, LPIPS from 93.81 to 70.18, MSE from 69.95 to 41.74, and SSIM from 85.09 to 87.51. Adding FRI with a naive addition (add) on top of FIJ gives Structure Distance 16.50, PSNR 25.93, LPIPS 85.44, MSE 38.72, SSIM 86.51. The full frequency-domain fusion (freq) design gives the best background preservation: 10.34, 27.32, 55.02, 28.66, and 89.21 respectively. The paper notes CLIP score alone does not reflect editing quality.

• Clinical augmentation (Table 4): On bleeding classification with ConvNeXt-T, all methods except ConvNeXt-T augment with roughly 5,000 additional images. FIA-Edit achieves the best AUC (85.05 vs 81.54 for the unaugmented ConvNeXt-T and 82.10 for traditional augmentation), PR-AUC (43.81), Precision (54.01), Recall (32.90), F1-score (40.89), and Accuracy (92.24). FlowEdit is second-best on PR-AUC (40.34, underlined), Recall (31.44), and F1 (38.86). MasaCtrl is second-best on AUC (84.22).

• Behavior of augmentation types: Traditional augmentation produced only marginal gains. The four inversion-based methods improved precision but degraded recall, which the authors suggest may stem from the limited generation quality of SD1.5. Both FlowEdit and FIA-Edit improved recall, but FlowEdit's weaker background preservation may cause false positives.

Methodology in Plain English

The method starts from the FlowEdit inversion-free paradigm. Instead of inverting the image to noise, it builds an intermediate noisy version of the source image, then estimates two velocity fields: one pointing back toward the source (using the source prompt) and one pointing toward the target (using the target prompt). The difference between these two fields gives the editing direction, and the edited latent is updated step by step.

The paper's addition is the FIA Constraint, which forces the source and target branches to share information while these velocity fields are being computed:

• Frequency Representation Interaction (FRI) runs inside the self-attention blocks. Source and target features are transformed with a 2D FFT, split into high- and low-frequency parts using a Gaussian low-pass filter, then recombined with weights of 0.8 and 0.2. The design emphasizes the source's high frequencies (fine texture and semantic detail) and the target's low frequencies, while suppressing the source's low frequencies and the target's high frequencies. The fused result is converted back with an inverse FFT and injected into the self-attention layers. The intuition is that structure and semantics disentangle more naturally in frequency space.

• Feature Injection (FIJ) runs inside the cross-attention layers of the later DiT blocks, specifically layers 13 through 23. It replaces the target branch's queries, keys, values, and text embeddings with the source-side ones. This is applied only during early generation steps, when the target and source latents are still similar, so the target absorbs source information smoothly under the target prompt's guidance.

Implementation settings: SD3.5-Medium with 50 sampling steps, classifier-free guidance scales of 3.5 for the source and 13.5 for the target. FRI is applied across all 50 steps; FIJ is activated only during the first 27 steps. During stepping, the reused interpolation noise is added with a scaling factor. Evaluation uses PIE-Bench (700 image–prompt pairs across 10 editing categories) and metrics including Structure Distance, PSNR, LPIPS, MSE, SSIM, and CLIP similarity on the whole image and edited regions; the dataset's region masks are used only for evaluation. The clinical dataset is the Laparoscopic Roux-en-Y Gastric Bypass dataset with 140 videos (100 training, 40 testing) sampled at 1 fps for 770K frames; the training set has 512K normal and 44K bleeding frames, the test set 197K normal and 17K bleeding frames, and 4,803 early-stage bleeding frames (about 50 per video) were edited.

Why This Matters

Impact on research: The paper argues that current inversion-free methods lack explicit source-feature integration, and demonstrates that combining frequency-domain fusion with attention-level feature injection can close much of the quality gap with inversion-based methods while remaining fast. It also broadens the scope of image-editing research by extending tuning-free editing to clinical data, claiming to be the first to do so.

Real-world applications:

• Medical data augmentation, particularly generating hemorrhage variations for surgical training and analysis where rare events cause severe class imbalance.
• Surgical video analysis and early bleeding detection support systems that help surgeons locate bleeding sites.
• General content creation and photo editing tasks such as object modification, addition, removal, color transformation, and text replacement.
• Visual effects production and social media content creation, both cited as existing application areas of generative editing models.

Industry relevance: The method runs at 6.30 s and 17.93 GB on a single RTX 4090, placing it in a practical range for consumer-grade hardware, whereas several compared baselines required an A100 80GB (up to 69.22 GB for RF-Inv). The claimed clinical results (AUC 85.05, F1 40.89) indicate potential value for medical software pipelines, and the code is released on GitHub.

Future Directions

• Determining whether task-specific finetuning of the editing model, rather than the untuned SD 1.5 or 3.5 checkpoints used here, would improve the recall degradation observed in inversion-based augmentation methods.
• Reducing the runtime gap with FlowEdit (6.30 s versus 3.49 s), which the paper attributes to the cost of feature interaction, while keeping the quality gains.
• Validating the medical editing approach on datasets and clinical tasks beyond the single Laparoscopic Roux-en-Y Gastric Bypass dataset used here, and quantifying the anatomical plausibility of synthesized bleeding variations.
• Exploring the FRI and FIJ design choices further, such as the fixed weighting coefficients of 0.8 and 0.2 and the decision to apply FIJ only in DiT layers 13–23 and only during the first 27 of 50 steps, which the paper does not ablate in the provided content.

Target Audience

Researchers and practitioners working on diffusion- and flow-based image editing, especially those focused on inversion-free or efficient editing pipelines; computer vision engineers who need controllable editing with high background fidelity on limited GPU memory; and medical imaging and surgical data researchers interested in generative data augmentation for imbalanced clinical datasets. The paper's heavy use of frequency-domain and transformer-internals terminology makes it most accessible to readers already familiar with latent diffusion architectures.

Authors’ abstract

Text-guided image editing has advanced rapidly with the rise of diffusion models. While flow-based inversion-free methods offer high efficiency by avoiding latent inversion, they often fail to effectively integrate source information, leading to poor background preservation, spatial inconsistencies, and over-editing due to the lack of effective integration of source information. In this paper, we present FIA-Edit, a novel inversion-free framework that achieves high-fidelity and semantically precise edits through a Frequency-Interactive Attention. Specifically, we design two key components: (1) a Frequency Representation Interaction (FRI) module that enhances cross-domain alignment by exchanging frequency components between source and target features within self-attention, and (2) a Feature Injection (FIJ) module that explicitly incorporates source-side queries, keys, values, and text embeddings into the target branch's cross-attention to preserve structure and semantics. Comprehensive and extensive experiments demonstrate that FIA-Edit supports high-fidelity editing at low computational cost (~6s per 512 * 512 image on an RTX 4090) and consistently outperforms existing methods across diverse tasks in visual quality, background fidelity, and controllability. Furthermore, we are the first to extend text-guided image editing to clinical applications. By synthesizing anatomically coherent hemorrhage variations in surgical images, FIA-Edit opens new opportunities for medical data augmentation and delivers significant gains in downstream bleeding classification. Our project is available at: https://github.com/kk42yy/FIA-Edit.