Instruction-based Image Editing: A Survey on Data, Models, Evaluation, and Applications

AI summary

Overview

• Research area: Computer Vision — generative image editing with natural-language instructions (Instruction-based Image Editing, IIE).
• Technical level: Intermediate (a survey that assumes familiarity with GANs, diffusion models, and vision-language models).
• Scope: A single survey that organizes IIE research around task definitions, training-data construction, model architectures, evaluation, and commercial products, and introduces a new benchmark called CDD-IIE Bench.

Note on completeness: the paper text available for this summary is truncated during Section 3 (data construction). Sections covering model surveys, the full CDD-IIE Bench results, commercial products, future directions, and conclusions are referenced in the abstract and introduction but their contents — including all numerical benchmark scores — are not present in the available text and are therefore not reported here.

What This Paper Is About

Instruction-based Image Editing lets a user change a photo by typing a sentence, such as "make the car red" or "remove the person on the left." Progress has been fast, but the field lacks a unified definition of what IIE tasks exist, a systematic review that covers newer autoregressive and hybrid architectures, and evaluation metrics rich enough to compare models fairly. This survey addresses all three gaps by proposing a task taxonomy, reviewing how training data is built, tracing model evolution, and introducing a new diagnostic benchmark.

Key Contributions

1. A systematic taxonomy with clear boundaries. The paper formally defines IIE as a transformation taking a source image and textual instructions as input and producing an edited image, then splits tasks into a basic atomic editing suite and an advanced compositional editing suite, covering 5 major evaluation dimensions and 21 specific editing tasks.
2. A comprehensive four-dimension survey. The review covers training-data construction methodologies, evolutionary trends in model architectures (GANs to diffusion to autoregressive to unified hybrid models), evaluation frameworks, and comparative assessment of open-source and proprietary solutions.
3. A hierarchical evaluation metric system. The authors propose a metric system with 5 principal categories and 21 fine-grained sub-criteria, paired with extensive benchmarking of cutting-edge models.
4. CDD-IIE Bench. A proposed benchmark — Comprehensive, in-Depth, and Diagnostic for IIE — intended to rigorously assess multiple aspects of model performance beyond existing metrics.

Main Findings

• Task taxonomy spans two suites. The basic atomic suite and advanced compositional suite together "comprehensively cover 5 major evaluation dimensions encompassing 21 specific editing tasks."
• Basic atomic editing covers 13 named operations. Object-level: Addition, Removal, Attribute Modification, Replacement, Text Modification, Portrait Enhancement, Motion Change, Extraction. Image-level: Tone Transformation, Style Transfer, Background Change. Utility: Image Repair, Visual Effect Removal.
• Advanced compositional editing covers 8 named operations. Complex Instruction & Reasoning: Parallel Instructions, Sequential Instructions, Implicit Reasoning. Spatial Understanding & Reasoning: Viewpoint Transformation, Referring-based Editing, Position Adjustment, Size Adjustment, Count Change.
• Architectural evolution runs through three paradigms plus hybrids. GANs handled coarse, domain-narrow edits (style transfer, face editing, image restoration, super-resolution, semantic manipulation); diffusion models added fine-grained control and text conditioning (Prompt-to-Prompt, DiffEdit, RePaint, Blended Diffusion); autoregressive models exploited strong language comprehension for finer-grained editing; unified models combining diffusion and autoregressive strengths followed.
• Removal data construction moved from implicit generation to modular pipelines. EmuEdit synthesizes the edited image first, then erases the object with attention-controlled masking. AnyEdit instead uses GroundingDINO and SAM to locate and mask targets, followed by Stable Diffusion inpainting; Step1X adds Qwen2.5-VL and the Flux Fill model.
• Addition is usually built as inverse removal. EmuEdit, AnyEdit, Step1X, and ImgEdit all construct add-task pairs by generating the edited image and then producing the source image through a removal instruction, avoiding manual annotation of new object masks but inheriting the accuracy of the removal pipeline.
• Compositional tasks are the weak point. For Parallel Instructions, the survey states that no existing dataset or model explicitly focuses on parallel instruction execution; methods execute edits sequentially or greedily rather than in one unified parallel pass. Sequential instruction datasets mostly support only single-turn edits and often lack semantic coherence between consecutive instructions, risking conflicting edits or semantic drift.
• Data filtering relies on similarity metrics. Removal and related pipelines are filtered using CLIP-based text-image similarity (instruction alignment), CLIP image-image similarity and L1 distance (preservation of unedited regions), and CLIP text-image similarity again to confirm the edit direction.
• Real-world degradation data supplements synthetic data for repair. Synthetic approaches use BSD500, GoPro, and a Watermark-Removal dataset; later efforts use RealBlur (burst photography), SIDD (real smartphone noise under diverse lighting), and CLWD (mixed synthetic and real watermarks).
• Specific construction parameters are reported for style transfer. EmuEdit's Plug-and-Play guidance produces ten stylized variants using DDIM inversion with guidance scales of 6.5–10.0 and structural blending ratios set to 0.8; AnyEdit applies the Prisma API to MSCOCO images with keyword-based filtering.
• Motion change uses video-derived supervision. Step1X selects frame pairs from Koala-36M using motion estimation with BiRefNet and RAFT, separates foreground from background, and has GPT-4o describe frame differences to generate fine-grained motion instructions.
• Benchmark scores are not available in the provided text. The abstract promises empirical comparisons of open-source models via CDD-IIE Bench, but no numerical results appear in the available content.

Methodology in Plain English

This is a survey paper, so its "method" is structured literature synthesis rather than new experiments. The authors first define IIE precisely and sort every editing operation they find in the literature into a two-level scheme: simple, single-purpose atomic edits versus composite edits that demand reasoning, multi-step planning, or spatial understanding. They then trace how training data for each task type is manufactured — showing that many tasks (addition, replacement, background change, extraction) are built "in reverse," where a model generates the edited image first and the original is reconstructed from it, because labeling the edit directly is hard. They organize models by generation: GANs, then diffusion models, then autoregressive models, then hybrid unified architectures. Finally, they argue existing metrics are insufficient and propose their own hierarchical metric system with 5 categories and 21 sub-criteria, along with CDD-IIE Bench, to test leading open-source models.

Why This Matters

Impact on research. The paper argues that the field suffers from three specific blockers: no unified task definition, no systematic review covering autoregressive and unified architectures, and insufficient evaluation metrics. By supplying a taxonomy, a data-construction map, and a 21-criterion evaluation scheme, it gives researchers a shared vocabulary and a way to compare models on more than a single similarity score.

Real-world applications (drawn from the paper):

• Product cataloging and visual asset reuse — the Extraction task isolates subjects onto clean backgrounds, which the paper explicitly frames as valuable for cataloging products and reusing visual assets.
• Modular image editing — extraction output is described as enabling modular editing workflows.
• Photo repair — Image Repair covers deblurring, denoising, and watermark removal.
• Adverse-weather image cleanup — Visual Effect Removal eliminates haze, rain, snow, and shadows, which the paper describes as supporting downstream vision applications in challenging environments.
• Consumer beautification — Portrait Enhancement covers skin refinement and makeup adjustments.

Industry relevance. The authors are all affiliated with the Institute of Artificial Intelligence (TeleAI), China Telecom, Beijing, and the survey devotes a dedicated section to mainstream commercial instruction-based editing products, positioning the work at the boundary between academic research and deployed "one-sentence photo manipulation" systems.

Future Directions

• Parallel instruction datasets and models. The paper explicitly identifies a gap: no existing dataset or model targets executing multiple independent instructions on one image in a single forward pass, calling it an opportunity for future research toward true compositional editing.
• Multi-turn, context-aware sequential editing. Better structured instruction chains are needed so that consecutive edits maintain semantic coherence instead of drifting or conflicting.
• Better handling of implicit instructions. Since implicit edits require commonsense and world knowledge rather than explicitly stated targets, the paper points to explicit intermediate reasoning — such as using LLMs to convert implicit instructions into explicit executable commands — as a promising direction.
• Stronger evaluation. The survey's stated motivation for CDD-IIE Bench is that existing metrics cannot comprehensively assess editing models, so improving and standardizing diagnostic evaluation remains an open problem.

Target Audience

Researchers and graduate students in computer vision and generative modeling who need a single organized entry point into instruction-based image editing; practitioners building or selecting editing models who want the data-construction recipes and the comparison of open-source versus proprietary solutions; and evaluation-focused engineers who need the taxonomy of 5 dimensions and 21 sub-criteria to design or interpret benchmarks. Readers without background in GANs, diffusion models, or vision-language models will find Sections 2 and 3 dense, since they include formal loss functions and detailed pipeline descriptions.

Authors’ abstract

Instruction-based Image Editing (IIE) aims to transform a given image into a new one based on textual instructions. Advances in Large Language Models (LLMs) and Vision-Language Models (VLMs) have accelerated progress toward practical ``one-sentence image editing" systems. This survey presents a systematic taxonomy and comprehensive review of IIE research, structured around five core dimensions: (1) task definition and hierarchical categorization of editing operations, (2) methodologies for training data construction, (3) architectural evolution from GAN-based to diffusion and autoregressive paradigms, (4) standardized evaluation metrics and benchmark development, and (5) introduction of commercial solutions. Our analysis shows critical technological milestones across model generations. We further propose a Comprehensive, in-Depth, and Diagnostic benchmark for IIE task (CDD-IIE Bench), which can rigorously assess the multiple aspects of model performance. Through empirical comparisons of open-source solutions, we highlight their respective capabilities and limitations. Finally, we discuss future research directions to advance the field.