SpatialReward: Bridging the Perception Gap in Online RL for Image Editing via Explicit Spatial Reasoning

AI summary

Overview

Research area: Computer vision, specifically reward modeling for instruction-guided image editing and its use as a training signal in online reinforcement learning (RL) for generative image models.

Technical level: Advanced. The paper assumes familiarity with multimodal large language models (MLLMs), chain-of-thought reasoning, reward modeling, and RL fine-tuning methods such as GRPO.

Scope: In one sentence: the paper diagnoses a failure mode it calls "Attention Collapse" in MLLM-based image-editing evaluators and proposes a reward model, SpatialReward, that uses explicit predicted bounding boxes to anchor cross-image verification, together with a 260k training set, a new multi-constraint benchmark, and online RL results on image editors.

What This Paper Is About

Instruction-guided image editors are increasingly aligned using online RL, but that approach depends on a reward model that can reliably judge whether an edit followed the instruction while preserving the untouched parts of the source image. The authors argue that existing evaluators have a perception gap: without explicit spatial guidance, their attention "collapses" away from the source image, so cross-image comparison degenerates into single-image judgment and scores drift from human preference. SpatialReward addresses this by forcing the reward model to first predict the edited regions and then reason about those regions explicitly.

Key Contributions

1. Identification of the perception gap. The authors characterize "Attention Collapse" in MLLM-based evaluators, showing that the absence of spatial anchors causes the model to neglect cross-image comparison and fine-grained detail, and they argue explicit spatial grounding is needed to fix it.
2. The SpatialReward framework. The paper presents what it describes as the first framework to integrate explicit spatial reasoning into generative pointwise evaluation for image editing, built on a "Think-with-Boxes" mechanism. It is released alongside SpatialReward-260k, a large-scale dataset of spatial reasoning traces.
3. MultiEditReward-Bench (MER-Bench). A new benchmark composed of complex multi-region edits, designed to stress the spatial perception and verification abilities of reward models.
4. Benchmark and online RL validation. Experiments report state-of-the-art results on MMRB2 and EditReward-Bench, outperformance of proprietary evaluators on MER-Bench, and significant downstream gains when the model is used as the reward signal for online RL.

Main Findings

• Gains over the generative baseline on public benchmarks: Compared with EditScore-8B, SpatialReward improves by +11.3% on EditReward-Bench (0.803 vs. 0.690) and +9.1% on MMRB2 (0.661 vs. 0.570). On EditReward-Bench it reports 0.683 on the PF metric and 0.672 on Consistency.
• Caveat on the discriminative baseline: EditReward attains a higher PF score (0.832) but its annotations cover only instruction adherence, so its consistency dimension is not evaluated (marked as "-" in Table 1). The paper argues this structural gap causes content drift and over-modification downstream.
• Cross-image generalization on MMRB2: SpatialReward reaches 0.608 on the Multi-Image subset and 0.671 on the Single-Image subset (overall 0.661), despite no specific training for that setting.
• MER-Bench results: SpatialReward achieves an Overall Accuracy of 48.3%, versus 35.0% for EditScore-8B and 46.2% for Gemini-2.5-Pro. In the hardest 4-Pair setting it reaches 21.5%, higher than Gemini-3.0-Flash (19.5%).
• Category behavior: Most models degrade on human-centric tasks; GPT-5 drops from 51.8% (General) to 30.0% (Human-Face), while SpatialReward holds 45.3% on Human-Face and 48.3% overall.
• Online RL improvement: Used as the reward signal, SpatialReward lifts OmniGen2 by +0.90 on GEdit-Bench and +0.28 on ImageEdit-Bench. For comparison, GPT-4.1 gives +0.45 on GEdit-Bench, EditScore +0.61, and EditReward +0.77.
• Transfer to a stronger editor: Applying the same pipeline to UniRef-Image-Edit raises source consistency from 7.81 to 8.02, GEdit-Bench from 7.46 to 7.56 (+0.10), and ImageEdit-Bench from 4.15 to 4.23 (+0.08).
• Efficiency: The paper reports a 1.5x inference speedup over EditReward through vLLM integration, and avoids the 4x inference averaging used by EditScore.
• Ablations on spatial grounding: Accuracy moves from 0.743 (SFT baseline without grounding) to 0.761 (SFT with box prediction only) to 0.778 (SFT with Think-with-Box) to 0.803 (with online RL).
• Ablations on aggregation: Weighted Geometric Mean scores 0.803, versus 0.774 for the Bucket Principle (Min) and 0.771 for Arithmetic Mean.
• Attention measurements (N=776 samples): The baseline shows a large entropy gap (3.48 ± 0.57), low source entropy (2.88 ± 0.71), high concentration (0.84 ± 0.05), and inter-sample correlation of 0.04 ± 0.18. SpatialReward shows gap 1.16 ± 1.10, source entropy 5.71 ± 0.81, concentration 0.37 ± 0.14, and correlation 0.12 ± 0.15.

Methodology in Plain English

The reward model is framed as a conditional generation problem: an input X (images plus instruction) is mapped to a structured output Y containing bounding boxes B, a textual rationale T, and scalar scores s.

Evaluation is split into two streams. The Semantic Consistency (SC) stream follows a "locate, then check" pattern: the model first predicts bounding boxes for the edited objects, then writes a rationale that cites box tokens to force a look back at the actual pixels, plus a global token to force a full-image scan, and finally emits instruction-following and source-consistency scores. The Perceptual Quality (PQ) stream is deliberately blind to the source image: it sees only the edited output and produces a reference-free rationale and naturalness/artifacts scores. This extends the VIEScore decomposition. Sub-scores are combined with weighted sums inside each dimension, and the two dimensions are combined by a geometric mean, R_final = (S_SC)^α · (S_PQ)^(1-α), with α = 0.80, w_SC = {0.6, 0.4}, and w_PQ = {0.5, 0.5}, tuned on a disjoint 2,000-sample validation set. Scores use the VIEScore range [0, 25].

The SpatialReward-260k dataset is built in three steps: a VLM (Qwen-3-VL-235B-A22B-Instruct) produces the bounding boxes; category-specific expert routing sends human-centric edits to Gemini-2.5-Pro with crop-focused prompts and general object edits to GPT-5 with bounding-box overlays to produce rationales and scores (PQ is evaluated separately by GPT-5); then an alignment and verification pass rewrites rationales into the interleaved format and discards samples where the rationale contradicts the visual evidence. The dataset combines 100k refined EditScore samples, 100k re-purposed EditReward samples, and 60k custom multi-edit samples.

Training is two-stage on a Qwen-3-VL-8B-Instruct backbone: supervised fine-tuning on the 260k samples, then group relative policy optimization (GRPO) on 7k mined hard samples, using Gemini-3.0-Flash as an online supervisor to produce consistency scores between 0 and 1. SFT uses LoRA (r = 32, α = 64) with learning rate 1e-4, 10 epochs, max length 8192, on 8 H800 GPUs; GRPO uses group size 4, KL coefficient 0.02, learning rate 5e-7, batch size 128, temperature 0.9, max length 1024, on 32 H800 GPUs.

For MER-Bench, instructions with 2–5 operations are generated across 15 subtasks, six outputs per instruction are sampled from 11 generation systems, and five trained annotators rate each along prompt following, perceptual quality, and overall aesthetics with majority voting. Final labels form 600 evaluation groups (1,800 samples): 200 two-pair sets, 200 three-pair sets (one Good, Medium, and Poor each), and 200 four-pair sets where the fourth sample is distinguishable only through fine-grained SC/PQ sub-dimensions.

Why This Matters

Impact on research. The paper reframes a reward-modeling problem as a perception-and-attention problem, arguing that the missing ingredient in pointwise MLLM judges is explicit spatial grounding rather than scale or more human labels. It provides quantitative attention evidence for that claim and a public benchmark aimed specifically at multi-constraint, fine-grained discrimination.

Real-world applications (as implied by the paper's setting and task taxonomy):

• Consumer photo editing tools, where a model must change only what the user asked and leave source identity intact (the paper's "source consistency" failure mode).
• Human-centric retouching such as pose, clothing, micro-expression, and texture edits, the category where general-purpose judges degrade most.
• Automated quality control and ranking pipelines for generative editing services, where an evaluator must convert scores into correct orderings.
• RL-based fine-tuning of editing models in production, where reward latency and reliability directly affect training cost and stability.

Industry relevance. The author list spans Kuaishou Technology, Harbin Institute of Technology (Shenzhen), Tsinghua Shenzhen International Graduate School, and HKUST, and the work is acknowledged as supported in part by Kuaishou Technology. The 1.5x inference speedup over EditReward and the avoidance of 4x inference averaging point to deployment concerns for online RL, and the UniRef-Image-Edit transfer test suggests the signal is not tied to one editor.

Future Directions

1. Region-aware reward modeling. The authors propose assigning distinct rewards to semantic editing regions, which would allow advantage estimation for individual edits within a single instruction — a form of region-level credit assignment intended to produce denser, localized supervision than a single scalar reward.
2. Denser credit assignment under FlowGRPO. The conclusion frames region-level rewards specifically as a complement to FlowGRPO, leaving open how such per-region advantages would be computed and combined.
3. Generalization beyond the tested backbones and datasets. The paper validates transfer on UniRef-Image-Edit, but whether the same spatial-prior pipeline holds for other editors, instruction styles, and languages is not reported.
4. Scaling the annotation pipeline. The dataset relies on expert routing among proprietary models (Qwen-3-VL-235B, Gemini-2.5-Pro, GPT-5) plus a Gemini-3.0-Flash supervisor during RL; how much of the performance depends on those specific teachers, and whether an open pipeline can match it, is not addressed in the provided content.

Target Audience

Researchers and engineers working on reward modeling, RLHF/GRPO-style alignment for generative models, and instruction-guided image editing, particularly those building fine-grained evaluators or training editors with online RL. It is also relevant to practitioners who need an offline judge for image-editing quality and to anyone studying attention behavior in multimodal models. Readers without background in MLLMs, chain-of-thought reasoning, or policy-gradient RL will find the method sections difficult.

Authors’ abstract

Online Reinforcement Learning (RL) offers a promising avenue for complex image editing but is currently constrained by the scarcity of reliable and fine-grained reward signals. Existing evaluators frequently struggle with a critical perception gap we term "Attention Collapse," where models neglect cross-image comparisons and fail to capture fine-grained details, resulting in inaccurate perception and miscalibrated scores. To address these limitations, we propose SpatialReward, a reward model that enforces precise verification via explicit spatial reasoning. By anchoring reasoning to predicted edit regions, SpatialReward grounds semantic judgments in pixel-level evidence, significantly enhancing evaluative accuracy. Trained on a curated 260k spatial-aware dataset, our model achieves state-of-the-art performance on MMRB2 and EditReward-Bench, and outperforms proprietary evaluators on our proposed MultiEditReward-Bench. Furthermore, SpatialReward serves as a robust signal in online RL, boosting OmniGen2 by +0.90 on GEdit-Bench--surpassing the leading discriminative model and doubling the gain of GPT-4.1 (+0.45). These results demonstrate that spatial reasoning is essential for unlocking effective alignment in image editing.