GRADE: Benchmarking Discipline-Informed Reasoning in Image Editing

AI summary

Overview

Research area: Computer Vision — image editing benchmarks for unified multimodal models, with a focus on discipline-informed (academic-domain) knowledge and reasoning.

Technical level: Intermediate. The evaluation protocol and the model taxonomy (unified multimodal models vs. specialized editing models, question-guided MLLM-as-a-judge) assume some familiarity with generative image editing and MLLM evaluation, but the main results are readable without deep technical background.

Scope: The paper introduces GRADE, a 520-sample benchmark spanning 10 academic disciplines, plus a three-dimension automated evaluation protocol, and reports results for 20 state-of-the-art open-source and closed-source image editing models.

What This Paper Is About

Existing image editing benchmarks mostly use natural images and instructions that require only everyday commonsense reasoning, so they do not really test whether a model can apply structured, domain-specific knowledge. GRADE ("Grounded Reasoning Assessment for Discipline-informed Editing") is the first benchmark built specifically to test discipline-informed knowledge and reasoning during image editing, covering domains from natural science to social science. The goal is to measure whether unified multimodal models can combine knowledge, reasoning, and precise visual modification rather than only producing visually plausible edits.

Key Contributions

1. The first discipline-informed image editing benchmark. GRADE contains 520 curated image editing triplets (input image, textual instruction, ground-truth image) spanning 10 academic disciplines, from natural sciences to humanities and applied fields.

2. A multi-dimensional automated evaluation protocol. The paper jointly assesses Discipline Reasoning (via weighted, question-guided MLLM judging), Visual Consistency (task-specific: localized, style, independence), and Logical Readability (clarity of structure, text, and annotation). The protocol is reported to align strongly with human judgments.

3. A large-scale empirical study. The authors evaluate 20 state-of-the-art models (10 closed-source and 10 open-source) on the benchmark, including both unified multimodal models such as GPT-Image-1.5 and Nano Banana, and specialized editing models such as Qwen-Edit and FLUX.

4. Diagnostic analysis and ablations. The paper analyzes representative failure cases, compares implicit versus explicit instruction formulations, and categorizes error types for a top-performing model to explain where current systems break down.

Main Findings

• Best model still fails most samples. Nano Banana Pro achieves the highest overall accuracy at 46.2%, followed by Nano Banana 2 at 39.6% and Seedream 5.0 at 24.7%. This means even the strongest model fails to satisfy all discipline-informed editing criteria in more than half of cases.

• GRADE sharply discriminates between models. Models that perform comparably on other benchmarks diverge substantially here: Nano Banana Pro at 46.2% versus GPT-Image-1.5 at 16.0% and Seedream 5.0 at 24.7%. The paper presents this as evidence of GRADE's stronger discriminative power for knowledge-intensive reasoning.

• Large closed-source versus open-source gap. The best open-source model, Qwen-Edit-2511, reaches only 2.7% accuracy, below every closed-source model. Several open-source models (OmniGen, Bagel, FLUX.2 dev) achieve near-zero or zero accuracy. The same gap appears under the relaxed score, where most closed-source models score above 40% while open-source models largely remain below that level.

• Reasoning is the bottleneck dimension. In Discipline Reasoning, Nano Banana Pro reaches 77.5%, ahead of Seedream 5.0 (64.1%) and GPT-Image-1.5 (54.5%), while the best open-source model, Qwen-Edit-2511, reaches only 18.6%.

• Consistency and readability act as constraints, not rankings. Many models score moderately on Visual Consistency and Logical Readability, so these dimensions mainly expose failure modes. The paper states FLUX.2 dev exhibits a notably low consistency score (17.6%), indicating instruction-scoped editing constraints are violated; the corresponding entry in Table 1 lists 56.4 for FLUX.2 dev. DreamOmni scores 83.2 on Visual Consistency and 89.1 on Logical Readability yet has an overall accuracy of 1.0, which the authors attribute to the model favoring minimal or no changes to preserve the original image.

• STEM disciplines differentiate models most; humanities remain hard. On Physics, Nano Banana Pro reaches 53.1% versus Seedream 5.0's 25.0% and GPT-Image-1.5's 15.6%. In Biology the gap is 55.6% versus 45.3% and 22.2%. History and Geography are difficult even for closed-source models, with Nano Banana Pro attaining only 29.6% on History.

• Explicit instructions help, especially open-source models. Converting implicit instructions into explicit ones raises Nano Banana 2's Discipline Reasoning from 67.9% to 89.7% and its accuracy from 35.3% to 65.7%. Qwen-Edit-2511 rises from 18.9% to 44.7% in reasoning and from 1.5% to 8.8% in accuracy, indicating open-source models rely more on explicit guidance.

• Automated judging aligns with human judgment, and Gemini-3-Flash is the best judge. On the 68-sample human alignment set, mean absolute error values across the three dimensions are around 10%. Gemini-3-Flash achieves the lowest MAE (0.1194 reasoning, 0.0954 consistency, 0.0838 readability) and the lowest standard deviation versus GPT-5 and Qwen3-VL-235B.

• Four recurring failure types. The error analysis of Nano Banana Pro identifies image recognition error (mis-parsed structural cues), knowledge error (failure to activate domain-specific priors), reasoning process error (correct methodology but flawed multi-step execution), and generation process error (correct planning but failure to enforce hard constraints during synthesis).

Methodology in Plain English

The authors first built a dataset of image editing triplets. For most of the data, six annotators with academic backgrounds sourced concept-grounded images from open textbooks, websites, and other reference materials, then manually created input and ground-truth image pairs and wrote the corresponding editing instructions, with two additional experts cross-validating. For the remainder, an automated pipeline coarsely filtered candidate samples from MMMU, after which two experts selected the final samples and designed the instructions, again followed by cross-validation by two additional experts.

The benchmark covers 10 disciplines (mathematics, physics, chemistry, biology, history, geography, sports, music, computer science, and economics), with a second-level sub-discipline taxonomy inside each, such as plane geometry, solid geometry, functions, graph, and statistics within mathematics. Every instruction is deliberately implicit: instead of describing the required operations step by step, it requires the model to infer them from disciplinary knowledge.

Evaluation proceeds along three dimensions. Discipline Reasoning is scored by generating weighted binary questions tied to the required knowledge, with weights summing to one, validated by two experts and cross-validated by a third; Gemini-3-Flash then judges the edited result against those scoring points and the ground-truth image, producing a normalized score between 0 and 1. Visual Consistency is scored 0/1/2 using prompts tailored to three task categories: localized consistency, style consistency, and consistency independence. Logical Readability is also scored 0/1/2, judging whether content is clear, logically consistent, and interpretable in academic terms.

Two aggregations are reported. Overall accuracy requires the maximum score on all three dimensions simultaneously. A relaxed score is a weighted average after normalizing each dimension to [0, 100], with weights of 0.6 for Discipline Reasoning, 0.3 for Visual Consistency, and 0.1 for Logical Readability.

Why This Matters

Impact on research. GRADE shifts image editing evaluation away from photorealism and everyday commonsense toward structured academic knowledge, exposing a capability gap that prior benchmarks, such as ImgEdit, RISEBench, and KRISBench, do not isolate. It also introduces a reusable question-guided judging protocol that the authors report aligns with five human experts' averaged scores at roughly 10% mean absolute error.

Real-world applications:

• Assisting researchers or teaching assistants in correcting geometric diagrams and textbook figures.
• Modifying chemical structures or reaction diagrams while preserving canonical notation such as bond-line representations.
• Refining data visualizations and charts so axes, curves, and legends remain interpretable.
• Filling in missing information in historical timelines or maps, where the model must infer the missing location and the underlying event.

Industry relevance. The measured gap between closed-source and open-source systems, and between top closed-source systems themselves (46.2% versus 16.0% versus 24.7% overall accuracy), identifies discipline-informed reasoning as a concrete target for model developers. The finding that explicit instructions substantially raise performance, especially for open-source models, also matters for product design: interface-level guidance can partially compensate for weaker implicit reasoning.

Future Directions

• Improve implicit reasoning, not just generation quality. Since explicit instructions lift Nano Banana 2 from 67.9% to 89.7% in Discipline Reasoning and Qwen-Edit-2511 from 18.9% to 44.7%, closing the implicit-to-explicit gap is a clear direction.

• Address the four error types. The taxonomy of image recognition, knowledge, reasoning process, and generation process errors points to distinct capability bottlenecks that could be targeted separately in training or inference.

• Strengthen humanities performance. History and Geography remain hard even for the strongest closed-source model (Nano Banana Pro reaches only 29.6% on History), raising the question of why non-STEM disciplines lag and whether different data or representations are needed.

• Extend and refine evaluation. The paper leaves open how the benchmark scales to more samples and disciplines, and how the judging protocol behaves as models improve. The reported difference between the prose (17.6% consistency for FLUX.2 dev) and Table 1 (56.4) also suggests a need for tighter reporting consistency in later versions.

Target Audience

Researchers and engineers working on unified multimodal models, image editing, and multimodal reasoning, who need a rigorous way to measure knowledge-grounded editing beyond visual realism. It is also useful for benchmark designers and evaluation researchers interested in question-guided MLLM-as-a-judge protocols, and for practitioners building educational, scientific, or technical tools that require precise, domain-correct image modifications.

Authors’ abstract

Unified multimodal models target joint understanding, reasoning, and generation, but current image editing benchmarks are largely confined to natural images and shallow commonsense reasoning, offering limited assessment of this capability under structured, domain-specific constraints. In this work, we introduce GRADE, the first benchmark to assess discipline-informed knowledge and reasoning in image editing. GRADE comprises 520 carefully curated samples across 10 academic domains, spanning from natural science to social science. To support rigorous evaluation, we propose a multi-dimensional evaluation protocol that jointly assesses Discipline Reasoning, Visual Consistency, and Logical Readability. Extensive experiments on 20 state-of-the-art open-source and closed-source models reveal substantial limitations in current models under implicit, knowledge-intensive editing settings, leading to large performance gaps. Beyond quantitative scores, we conduct rigorous analyses and ablations to expose model shortcomings and identify the constraints within disciplinary editing. Together, GRADE pinpoints key directions for the future development of unified multimodal models, advancing the research on discipline-informed image editing and reasoning. Our benchmark and evaluation code are publicly released.