Aphanta: Diagnosing Task-Aligned Image-Edited Intermediates for Multimodal Reasoning

AI summary

Overview

Research area: Multimodal reasoning and generative image editing — specifically, whether an MLLM can call an image editor to produce an explicit visual intermediate (an edited RGB image) and then reason over that intermediate.

Technical level: Advanced. The paper assumes familiarity with multimodal large language models (MLLMs), instruction-conditioned diffusion image editors, LoRA fine-tuning, and evaluation protocols such as accuracy and IoU scoring.

One-sentence scope: Aphanta is an automated task-discovery and closed-loop diagnostic framework that compares direct reasoning, editor-generated-intermediate reasoning, and idealized-reference-intermediate reasoning across 20 candidate tasks to map when image-edited intermediates actually help a multimodal pipeline.

What This Paper Is About

Multimodal models can be prompted to transform an image (crop, zoom, mark, sketch, delete an object) so that task-relevant evidence becomes easier to use, forming what the authors call a "visual workspace." The problem is that a visually plausible edit may still fail the exact count, symbol, relation, or counterfactual state a reasoning task requires, so an image editor that "looks right" can still hurt the final answer. Aphanta measures this by running the fixed chain MLLM → image editor → MLLM under three conditions (Direct, Actual Edit, Reference) to separate potential visual headroom from the practical utility of today's editors.

Key Contributions

1. A reusable diagnostic framework. Aphanta is a task-discovery and validation framework for measuring visual headroom and practical utility inside an MLLM–image-editor–MLLM loop, built around a three-condition A/B/C comparison.

2. A 20-task diagnostic study plus datasets. The authors construct Aphanta Train and Aphanta Test and report a 20-task audit spanning cue injection, grounding, counterfactual state realization, and structured visual construction — deliberately retaining filtered and unsuccessful tasks rather than only successful demonstrations.

3. A task-conditioned reliability boundary. Across multiple MLLM–editor combinations (and two external benchmarks, which are referenced but not named in the provided text), the authors identify where editors help and where they fail, and translate this into guidance for selective, verifiable visual assistance.

4. An explicit scope statement about causality. The paper delimits what the A/B/C protocol can and cannot establish, noting that it measures pipeline-level utility rather than the causal contribution of the returned pixels.

Main Findings

• Utility is strongly task-conditioned. Gains concentrate in visual cue injection, grounding, and counterfactual state realization, while intermediates requiring symbol-sensitive construction or structural extrapolation are substantially less reliable.

• The consolidated Qwen pipeline improves the selected positive-task subset. Mean task score rises from 0.343 to 0.445, an absolute change of +0.102 (+10.2 points) and a +29.7% relative gain. Its reference score of 0.558 indicates remaining realization headroom.

• Cross-model results vary widely. For complete within-family pipelines on the selected subset: Qwen3-VL + Qwen-Image-Edit moves 0.343 → 0.445 (+10.2 points); Seed-2.0 + Seedream-4.5 moves 0.505 → 0.540 (+3.5); GPT-5 + GPT-Image-1.5 moves 0.475 → 0.425 (−5.0); Gemini-3 + Nano Banana 2 moves 0.580 → 0.625 (+4.5).

• Holding the reasoner fixed, the editor alone changes the outcome. With Qwen3-VL fixed, editor deltas range from −4.7 to +10.2 points: Qwen-Image-Edit 0.445 (+10.2), GPT-Image-1.5 0.430 (+8.7), FLUX.2 Klein 0.380 (+3.7), Nano Banana 2 0.370 (+2.7), LongCat-Image-Edit 0.365 (+2.2), Seedream-4.5 0.296 (−4.7).

• Macro-averages by dominant operation (Table 2). Grounding: Direct 0.673, +Edit 0.773, +Ref 0.758 (edit Δ +0.100, ref Δ +0.085). Cue injection: 0.195 / 0.365 / 0.520 (edit Δ +0.170, ref Δ +0.325). State realization: 0.283 / 0.580 / 0.813 (edit Δ +0.297, ref Δ +0.530). Structured: 0.473 / 0.404 / 0.556 (edit Δ −0.069, ref Δ +0.083).

• Counterfactual state realization is the clearest positive region. Editing clocks, deleting specified objects, or completing a masked state increases the downstream score by 0.21–0.37.

• Task outcomes are graded, not binary. Four tasks — Repeated Pattern Recognition, Visual Equation Puzzle, Gear Rotation Reasoning, and Spot-the-Difference (Sparse) — were stopped after preliminary diagnosis because of limited reference headroom, a saturated baseline, unavailable edited outputs, or insufficiently stable construction. Of the 16 tasks taken forward, 15 reached final-stage evaluation (Circuit Diagram Parsing retains its Phase-2 result), and 13 yielded a positive retained pipeline. Plane Geometry Auxiliary Line and Flowchart Decision did not provide reliable final-stage assistance.

• Positive examples from the audit (Direct → +Edit → +Ref). Analog Clock Reasoning 0.50 → 0.81 → 0.84; Counterfactual: Deletion 0.10 → 0.47 → 0.92; What Is Behind the Mask? 0.25 → 0.46 → 0.68; Dense Dot Counting 0.05 → 0.23 → 0.52; Tangram 0.31 → 0.51 → 0.61; Industrial Defect Inspect 0.70 → 0.82 → 0.88; RPM: Real 0.61 → 0.79 → 0.71; Spot-the-Diff (Dense) 0.18 → 0.46 → 0.68.

• Negative and mixed cases. Gear Rotation Reasoning drops from 0.92 direct to 0.42 with edit (reference 0.83); Circuit Diagram Parsing drops from 0.70 to 0.60 (reference 0.88); Flowchart Decision drops from 0.36 to 0.22 (reference 0.41); Visual Equation Puzzle is 0.85 direct and 0.85 reference with no edit score reported.

• Three empirical findings explain the boundary. (1) Reliable visual-state realization — editors help most when the requested change is a local addition, deletion, attribute change, or perceptual cue that preserves the rest of the scene. (2) Plausibility can exceed task fidelity — an intermediate can look coherent while violating a count, symbol, correspondence, or geometric constraint; in the illustrated numbering case, sequential labels are rendered naturally but one item is skipped, and the error propagates downstream. (3) Structured extrapolation is less reliable — exact symbol recognition, logically valid construction, and topology-preserving routing are inconsistent, though the two RPM variants improved after task-specific development, so a blanket claim of "no visual logic" is not supported.

• C is a diagnostic target, not a strict ceiling. The actual edit occasionally exceeds the constructed reference, which is why the authors call condition C a reference rather than a strict upper bound.

Methodology in Plain English

The authors define one fixed inference chain: given a question and an image, an MLLM writes an edit instruction, an image editor produces an edited image, and an MLLM then answers from the original image plus the edited intermediate. For every candidate task they score three versions — A (Direct), answer from the original image alone; B (Actual Edit), the full editor-in-the-loop pipeline; and C (Reference), answer from a programmatically constructed "ideal" intermediate. The gap C − A is read as diagnostic visual headroom, and B − A as practical utility of the real pipeline; a large positive C − A with a small or negative B − A signals a realization gap.

To scale discovery, they built an agent loop with four phases: propose and screen candidate tasks; run a small preliminary A/B/C diagnosis; construct and human-curate data; then train and evaluate the editor-in-the-loop pipeline, feeding results back into the next round of proposals. A task-development agent (GPT-5.3-Codex) proposes and implements task assets, but all quantitative scores come from the fixed inference pipelines, not from the agent.

Data details: training rows for Qwen-Image-Edit are normalized to the CSV fields edit_image, image, and prompt; a strict-schema raw export contains 186,548 rows, and a balanced export resamples each pipeline to a 10k-row target (constrained by an 8k–12k policy), yielding 190,000 rows from 19 training pipelines. For evaluation, the full joint set contains 2,028 rows and the lite set contains 220 rows (20 examples from each of 11 completed pipelines); the full set is smaller than 11 × 200 because the real Raven split contributes 28 standard test examples. LoRA training uses AdamW with learning rate 1×10⁻⁴, rank 32, 20k–50k steps depending on data volume, on nodes with 8 NVIDIA H100 GPUs. At inference the pipeline averages roughly 1.7 editor calls per evaluated sample, and the joint evaluator's default maximum number of planned edit steps is two. Accuracy tasks use normalized exact match; localization tasks use a task-specific IoU scorer; each sample writes per-step traces, predictions.jsonl, metrics.json, and failure_cases.jsonl. Human reviewers check feasibility, measurability, reference leakage, and train/eval separation.

The authors state four limitations explicitly: the discovery process is exploratory, so Phase-2 selection can inflate gains on the retained subset; operation labels overlap and a larger preregistered universe is needed before treating macro-averages as population estimates; closed-source editors hide architecture and decoding details, so cross-family results characterize products and APIs at evaluation time; and A/B/C establishes pipeline-level utility but not the causal contribution of the returned pixels, since condition B also includes the edit instruction and an extra MLLM turn.

Why This Matters

Impact on research. The paper reframes the debate from "can diffusion models reason?" to "how well do a task, an intermediate representation, and an editor align?" That shift matters because it makes negative and stopped tasks part of the record, and it separates an editor's rendering ability from the downstream model's ability to use evidence. The diagnostic protocol is offered as reusable infrastructure rather than a single leaderboard.

Real-world applications examined in the task pool:

• Automotive assistance, via the Auto. Driving Assistant grounding task (direct 0.42, +Edit 0.57, +Ref 0.45).
• Industrial quality control, via the Industrial Defect Inspect grounding task (direct 0.70, +Edit 0.82, +Ref 0.88).
• Diagram and document decision support, via flowchart and circuit-diagram tasks — both of which exposed editor failures, showing where automation should not be deployed unchecked.
• Accessibility and visual question answering, via zoom-in/remake, mirror localization, and masked-state completion, where isolating or reconstructing a region is the requested operation.

Industry relevance. The evaluation spans commercial editors and models (GPT-Image-1.5, Nano Banana 2, Seedream-4.5, FLUX.2 Klein, LongCat-Image-Edit, Qwen-Image-Edit, GPT-5, Gemini-3, Qwen3-VL, Seed-2.0), and the results show that swapping only the editor shifts mean task scores by −4.7 to +10.2 points with the reasoner held fixed. For teams building agentic or tool-using multimodal products, that argues for routing visual assistance by task, verifying the generated intermediate, and rejecting or replacing it when unreliable — rather than turning on editing unconditionally.

Future Directions

1. Add the causal controls the authors say are missing. Matched second-pass reasoning, call-without-return, identity or sham images, and step-level counterfactual replacements are named as necessary to separate instruction scaffolding, editor realization, and downstream evidence use.

2. Preregister a larger task universe. The current 20-task pool is exploratory; a broader, preregistered suite is needed before the operation macro-averages can be read as population estimates.

3. Improve editors on symbol-sensitive and structural operations. Circuit symbols, geometry constructions, gear relations, and flowchart routing show positive reference headroom but weak or harmful actual edits — the signature of a task–editor mismatch that targeted training might close, as prior targeted work on maze planning and Sudoku suggests.

4. Build selective, verifiable visual assistance. The paper's own conclusion calls for systems that choose visual assistance by task, verify it after generation, and reject it when the evidence is unreliable.

Target Audience

Researchers and engineers working on multimodal reasoning, visual chain-of-thought, and generative image editing — particularly those deciding whether to add an editor call to a reasoning pipeline. It is also useful for benchmark designers, since the A/B/C protocol and the retained negative results show how to report a task-conditioned utility map instead of a success-only collection, and for product teams evaluating commercial image editors as components of an agentic system.

Authors’ abstract

Explicit visual intermediates can help multimodal large language models (MLLMs) externalize spatial evidence and updated visual states, but their utility depends on whether an image editor can faithfully realize the required transformation. We introduce \textbf{Aphanta}, an automated task-discovery and closed-loop diagnostic framework for the MLLM -> image editor -> MLLM pipeline. Aphanta evaluates three conditions---direct reasoning, reasoning with an editor-generated intermediate, and reasoning with an idealized reference intermediate---to separate potential visual headroom from the practical utility of current editors. Across 20 candidate tasks and multiple editor--MLLM combinations, we find that utility is strongly task-conditioned. Gains concentrate in visual cue injection, grounding, and counterfactual state realization, whereas intermediates requiring symbol-sensitive construction or structural extrapolation are substantially less reliable. On the selected positive-task subset, our consolidated Qwen pipeline improves the mean task score from 0.343 to 0.445 ($+10.2$ points; $+29.7\%$ relative), while the full study also retains filtered and unsuccessful tasks to expose the boundary. These results position image editing as a specialized visual workspace rather than a universal reasoning mechanism, and establish Aphanta as a reusable protocol for measuring task--representation alignment, editor realization, and downstream pipeline utility.