EditAgent: Rethinking Agentic Image Editing

AI summary

Overview

Research area: Computer Vision — agentic (multi-step, tool-using) instruction-based image editing and VLM-based evaluation of edited images.

Technical level: Advanced. The paper assumes familiarity with diffusion-based instruction editors, agent planning loops, vision-language model judging, and reward-model benchmarking.

Scope: The paper diagnoses two bottlenecks of agentic image editors — the set of tools the agent can act with and what its judge can observe — and introduces EditAgent, a system that widens both, together with a 600-set expert-annotated diagnostic benchmark and experiments on three editing benchmarks.

What This Paper Is About

Real editing requests are compound ("move me left, widen the shot, remove the people behind me, make it sunset"), so agentic editors split them into steps and run a plan–edit–judge loop. When a step fails, these agents mostly retry with the same single general-purpose editor and a judge that scores one output by eye from just the source image, the edited image and the instruction. The paper argues this is bounded by two spaces — the agent's action space (what it can do) and the judge's observation space (what it can see) — and builds a system that expands both.

Key Contributions

1. Diagnosis of two bottlenecks. Using a new expert-annotated benchmark of 600 candidate sets over 30 edit types, the authors show that a general-purpose editor produces no usable result in 42.7% of sets, and that no single judge (including finetuned reward models) is the most accurate across edit groups.
2. Expanded action space. Each step is routed to a pool that mixes a general editor with specialized methods: masked local editing and recoloring (inside a SAM 2.1 mask), object removal with OSOR, object movement, subject scaling by segmentation and compositing, global and local photometric operators, style filters and neural style transfer, flips/rotations/rotation-to-level, and zoom-in via cropping with super-resolution or zoom-out via outpainting.
3. Expanded observation space. Every candidate carries a verification record — change maps, close-ups of the edited region, measured color and geometry, and programmatic check outcomes — and all candidates are compared in a single listwise VLM call. With records, off-the-shelf VLM judges beat finetuned reward models without any finetuning.
4. Parallel search instead of sequential refinement, plus a demonstration that EditAgent works both as a general editing framework (improving two backbones on three benchmarks) and as a data engine whose 25.7K accepted sessions train stronger standalone editors.

Main Findings

• The general editor is not enough, and more sampling does not fix it. Of the 600 sets, the winner set contains only general-editor candidates in 254 (42.3%), only specialized candidates in 175 (29.2%), both kinds in 90 (15.0%), and nothing usable in 81 (13.5%). The general editor fails in 256 sets (42.7%), and a specialized method recovers 175 of those 256 failures.
• Failures are concentrated by edit group. Winner sets contain only general-editor candidates in 73.8% of Style & Effects sets, but only specialized candidates in 72.5% of Enhancement and 66.7% of Geometry sets.
• Extra samples from one editor lose to a wider pool. Comparing a pool of parallel general-editor samples against a pool that also includes specialized methods, with the same VLM judge selecting the delivered edit, the delivered edit is usable in only 45.2% of sets for the general-editor pool versus 68.2% with specialized methods. In the 296 sets where exactly one delivered edit is usable, it comes from the specialized pool in 217.
• Parallel search beats sequential refinement. On 144 Complex-Edit samples at complexity levels 1, 4 and 8, five professional Photoshop editors preferred parallel search in 102 comparisons and sequential refinement in 28, with 14 ties — 78.5% of the 130 decided comparisons — at no more than half the judge calls (at most 2 VLM calls versus 4).
• No judge wins everywhere. EditScore-72B has the best overall Winner-Set Hit@1 but is best only on Scene; Qwen3.8-27B, most accurate on three groups, is only 5.2 points above random on Photometric. CLIP-I is best on Photometric and Enhancement but at or below random on Style & Effects and Geometry.
• Off-the-shelf VLMs beat finetuned reward models once they can compare. On EditReward-Bench, EditScore models reach 65.9–70.3%, and EditScore-72B reaches 76.3% with four-pass ensembling. Pointwise, off-the-shelf VLMs reach 71.7–73.8%; listwise, 85.2–86.1% (Qwen3.5-27B 71.7 → 85.2, Qwen3.8-27B 72.1 → 85.9, Gemma-4-31B 73.8 → 86.1).
• Verification records help most where comparison hurts. On the 515 diagnostic sets with at least one usable edit, the Qwen3.8-27B judge scores 71.6% pointwise and 73.6% listwise, but listwise alone drops Photometric from 54.2% to 47.2% and Enhancement from 65.0% to 36.8%. Adding records recovers them to 64.4% and 80.3% and raises the overall score to 81.3%, 6.5 points above EditScore-72B.
• The effect generalizes across judges. Seven off-the-shelf VLM judges improve by 6.3 to 9.3 points with records; without records none reaches EditScore-72B (74.8%), and with records every one exceeds all four finetuned reward models, with Qwen3.8-Flash-Next reaching 82.8% (8.0 points above EditScore-72B).
• Multi-turn editing improves on EdiVal-O. Qwen-Image-Edit-2511 goes 75.97 / 58.58 / 47.64 to 80.64 / 67.02 / 55.46 (turns 1/2/3); FLUX.2-klein-9B goes 81.37 / 66.30 / 53.20 to 84.46 / 71.10 / 59.45. At 1024 px, Qwen-Image-Edit-2511 rises from 59.80 to 62.58 at turn 3.
• Complex instructions improve too. ComplexBench-Edit average rises by 6.43 points (54.40 → 60.83) for Qwen-Image-Edit-2511 and 3.06 points (61.45 → 64.51) for FLUX.2-klein-9B, improving all five groups; Complex-Edit overall rises by 0.074 and 0.036. Specialized methods deliver 19% to 31% of accepted steps.
• It beats existing agentic editors. With the same backbones and VLM, EditAgent takes the best turn-3 score (59.45 for FLUX.2-klein-9B versus 58.42 for the strongest agent, RISE-Agent) and the best score in all three turns, at 12.9 and 6.3 H100 GPU hours versus 1.1 and 0.5 for standalone editing.
• Accepted sessions make good training data. Finetuned on ~25.7K EditAgent sessions (about four turns each), Qwen-Image-Edit-2511 gains 1.95 points on EdiVal-O turn 3 and 3.17 on ComplexBench-Edit; FLUX.2-klein-base-9B gains 2.00 and 2.14. Data generated directly by FLUX.2-klein-9B degrades FLUX.2-klein-base-9B on all three benchmarks, whereas EditAgent data improves it.
• Ablation confirms each piece matters. On EdiVal-O turn 3 with Qwen-Image-Edit-2511: vanilla agent 47.64 → 49.43; adding the expanded action space +2.94 points → 52.37; adding parallel comparison → 53.18; adding verification records → 55.46 (+3.09 for the observation-space expansion). Replacing the action space with K identical copies of the general editor gives 47.53 (K=2), 48.90 (K=4), 49.01 (K=8).

Methodology in Plain English

The authors first build a diagnostic benchmark: 600 editing requests covering 30 fine-grained edit types in six groups (Content, Style & Effects, Scene, Photometric, Geometry, Enhancement), drawn from COCO val2017 images with 20 requests per type. Candidates (2–4 per set, 1,995 in total) come from two EditAgent runs — one with the full tool pool, one restricted to the general editor FLUX.2-klein-9B. Five professional Photoshop editors label each set blind, under neutral names C1–C4, selecting every usable candidate (the "winner set"). The metric is Winner-Set Hit@1: how often a judge's top pick is usable, computed on the 515 sets that have at least one usable candidate.

They then run three targeted studies on this benchmark: whether more samples from one editor suffice, whether to refine sequentially or search in parallel under a fixed budget of four generations, and whether any one-at-a-time judge is reliable.

EditAgent itself is a pipeline. A planner splits the request into ordered steps. For each step, an analyzer names the operation and target, a grounder localizes the target with a VLM plus SAM 2.1 when a mask is needed, and a router picks which methods apply. Those methods each generate at least one candidate in parallel from the same source image, each carrying a verification record: a change map, zoomed close-ups of the edited region, measured brightness/clipping/color shift, and outcomes of programmatic checks, alongside a note of what is left for visual judgment. One off-the-shelf VLM then sees the instruction, source image, all candidates (in random order) and all their records in a single listwise call, and either accepts one candidate or returns feedback for a retry. Each accepted image becomes the source of the next step, and a history check shows the VLM the original image, earlier turn results and the current image to catch a later edit undoing an earlier one, triggering a repair turn adopted only if the step judge accepts it. Qwen3.8-27B fills the planner, analyzer, grounder and judge roles; each attempt allows up to three candidates, each step two retries, and each turn two plan revisions.

Why This Matters

The paper reframes a practical failure mode: agentic editors that "mostly retry" are limited less by model quality than by tool diversity and by what the judge is shown. It offers a concrete, model-agnostic recipe — route to specialists, search in parallel, and hand the judge evidence — that works without finetuning any judge, and it shows the system can bootstrap better standalone editors from its own accepted sessions.

Real-world applications:

• Consumer and professional photo retouching apps that must honor long, multi-part natural-language requests.
• E-commerce and catalog imagery, where precise geometric and photometric operations (extend canvas, rotate, adjust brightness/contrast) are common and where a general editor alone failed most often.
• Marketing and social content pipelines that need compound edits (background replacement, time-of-day change, style application) with verifiable quality gates.
• Training-data production: using accepted sessions as supervision to improve smaller, cheaper editing models.

Industry relevance: The work comes from Adobe Research (with UC San Diego), targets the tooling layer rather than a single model, and is demonstrated on top of two open-weight backbones, Qwen-Image-Edit-2511 and FLUX.2-klein-9B — so it can be layered onto existing editors. Its emphasis on programmatic verification and auditable evidence also fits production settings where each edit needs a checkable record.

Future Directions

• Cutting the cost. EditAgent takes roughly 10× the inference time of standalone editing models (12.9 and 6.3 H100 GPU hours on EdiVal); the authors suggest amortizing this by distilling into stronger, more efficient standalone editors via the data engine.
• Widening the capability pool further. Performance is explicitly bounded by the underlying editing methods, so adding or improving specialists (especially for Geometry and Enhancement, where the general editor failed most) is an open lever.
• Richer, more automatic verification records. Records currently mix close-ups, measurements and code checks; which checks matter for which edit type, and how to generate them without hand design, is left open.
• Scaling the diagnostic benchmark. The current study spans 30 edit types and 600 sets with 515 usable for Hit@1; extending to more edit types, harder compositional requests, and unbiased judges (the authors already add Gemma-4-31B as an alternative internal judge to check for self-bias) remains future work.

Target Audience

Researchers and engineers working on instruction-based image editing, multimodal agents, and VLM-as-judge evaluation; practitioners building production editing pipelines who need reliable multi-step execution; and teams curating training data for generative image models. Readers should be comfortable with diffusion editors, agent loops, and benchmark design — the paper is written at an advanced level and assumes that background.

Authors’ abstract

Agentic image editors decompose complex requests into steps, but when a step fails, they mostly retry with the same narrow set of tools and a judge that scores each output by eye directly. We argue that such agents are bounded by two spaces: the action space of what the agent can do, and the observation space of what its judge can see. On our expert-annotated benchmark, a general-purpose editor yields no usable result in over 40% of the cases, and no judge is reliable across edit types. We present EditAgent, which expands both spaces. It routes each step to general editors and specialized methods, and attaches to each candidate a verification record. With these records, off-the-shelf VLM judges outperform finetuned reward models. Candidates are generated in parallel and judged in one listwise call: experts prefer this to sequential refinement in 78.5% of decided comparisons, at no more than half the judge calls. EditAgent improves two open-weight editors on three benchmarks and serves as a data engine for training stronger editors.