Region-Level Policy Optimization for Fine-grained MLLM Perception

AI summary

Overview

Research area: Computer vision and multimodal large language models (MLLMs), specifically efficient fine-grained visual perception and region-of-interest (RoI) prediction.

Technical level: Intermediate to Advanced. The paper assumes familiarity with MLLM architectures, visual token budgets, attention maps, and reinforcement learning terminology (policies, advantages, credit assignment).

Scope: One sentence: The paper shows that localizing answer-relevant evidence tolerates far more token compression than recognizing its content, and turns that asymmetry into a region-level reinforcement learning recipe that trains a lightweight RoI predictor using only a frozen MLLM reader's answer likelihood.

What This Paper Is About

Multimodal LLMs handle fine-grained detail poorly — small text, distant objects, cluttered high-resolution scenes — and the standard fix is to raise input resolution, which adds visual tokens and inflates both vision-encoding and language-model prefilling costs. The paper observes that fine-grained question answering actually involves two separate operations, localizing the region of interest and recognizing its content, and that these two operations have different resolution requirements. The goal is to exploit that difference: localize from a coarse view, then spend resolution only on the selected evidence, using a fast RoI predictor that is trained to be faithful to the answer rather than to a noisy attention surrogate.

Key Contributions

1. A controlled resolution diagnostic that isolates localization from recognition and reveals their asymmetric compression tolerance. Built on ZoomBench with Qwen3.5-4B, under a cap of at most 128 visual tokens, the localization sweep compresses only the scene from which the box is re-predicted while the recognition sweep freezes the box and compresses its crop. Localization tolerates roughly 3–4 times stronger token compression than recognition at matched survival, and the box is still re-predicted correctly at 25 times token reduction while reading the crop fails at 9 times.

2. Region-level policy optimization (Vision-RL²), a method that operates natively on a dense RoI map and trains a decode-free predictor from a frozen reader's functional answer signal. It treats coherent regions as actions and scores each region by how its removal changes the teacher-forced likelihood of the gold answer, with a subtractive objective (pruning regions whose contribution falls below a calibrated noise margin) and an additive objective (recovering evidence the proposal missed). It requires no region annotations, no response sampling, and no reasoning trajectories.

3. Sparse visual encoding, an inference scheme that takes the bounding box of the predicted foreground, measures foreground occupancy, and re-encodes only foreground tokens at finer spatial resolution while background tokens never enter the vision encoder or the language model.

4. Broad empirical validation across six fine-grained benchmarks and four MLLM backbones, showing improvements over both the base model and the supervised SD-RPN predecessor at every shared token budget, competitive results against full-finetuning state-of-the-art methods while training an order of magnitude fewer parameters. Code is available at https://github.com/YuHengsss/VisionRL2.

Main Findings

• Localization and recognition have different resolution requirements. In the controlled diagnostic on ZoomBench with Qwen3.5-4B (population survival n = 238), localization tolerates roughly 3–4 times stronger token compression than recognition at matched survival. The box is still re-predicted correctly at 25 times token reduction, while reading the crop fails at 9 times.

• Region-level RL improves fine-grained accuracy at every token budget. Across six fine-grained benchmarks and four MLLM backbones, Vision-RL² improves accuracy over the base model at every token budget and surpasses the base model's largest-budget accuracy with about 4 times fewer visual tokens.

• Strong main-protocol results. Under a shared 16,384 source-image token limit, Vision-RL² on Qwen3.5-9B attains the highest average in Table 1 at 80.1, ahead of Gemini-3.1-Pro (79.3) and Vision-OPD-9B (78.7), with the best V* Bench (95.3) and ZoomBench (68.4) scores overall. On Qwen2.5-VL-7B, Vision-RL² reaches the highest average of its block at 71.0, ahead of ZwZ-7B's 69.9. On Gemma-4-12B the predictor adds 3.8 points over SD-RPN and 9.5 over the base model on average.

• A deliberate trade-off at the 4B scale. Vision-RL²-4B surpasses Vision-OPD-4B on V* Bench, ZoomBench, and both HR-Bench splits, but trails on the MME-RealWorld splits, where full fine-tuning can adapt the reader itself — a gap the paper attributes to its frozen-reader design.

• Large efficiency gains. Under the training-aligned protocol, the model at the 576-token limit exceeds the base model at the 4,096-token limit by more than three points while consuming about one quarter of the tokens. It matches SD-RPN's 4,096-limit accuracy with 4.2 times fewer visual tokens on the 4B model and comes within half a point with 2.5 times fewer on the 9B model. In latency, the 4B model at the 1,024 limit exceeds SD-RPN at 4,096 with 2.1 times lower latency, and both models at the 576 limit surpass the base model at 4,096 at 2.3 times lower latency.

• Routing is much cheaper than alternatives. Attention routing and coordinate decoding spend a full source pass plus box tokens purely on localization, taking 1.1–1.8 s per query, or 2.1–2.2 times Vision-RL²'s end-to-end latency. The RPN head adds three blocks on the answer call's own prefill and routes in 31–49 ms, roughly 30 times cheaper, while reaching the best accuracy at both scales.

• Both mechanisms contribute. Ablations on Qwen3.5-4B at the 576-token source limit show region-level RL alone contributes +3.0 average over SD-RPN and the sparse visual encoding adds a further +1.5. Relative to the frozen base model, the full system gains +14.9.

• Design choices matter measurably. Replacing the functional score with generation accuracy costs −2.3 (binary correctness gives no relative credit when all actions share an outcome); raw mean log-probability costs −1.5; cell-level keep-sets instead of coherent regions cost −1.2; removing the control-region margin costs −1.1; removing the additive recovery group costs −0.9. Adding pair removals performs at parity with the full recipe at clear extra cost, so leave-one-out is kept.

Methodology in Plain English

The authors start from a diagnostic experiment. They take a model that predicts a box around the relevant region and then answers from that crop, and they compress visual tokens in two separate ways: one sweep compresses only the scene the locator sees, the other freezes the box and compresses the crop the recognizer sees. Tracking how often answers survive shows the locator is far more robust to compression than the recognizer.

That gap motivates decoupling the two: predict the region from a coarse view, then spend resolution on the selected evidence. The bottleneck becomes the RoI predictor, which must be reliable from a coarse view and cheap to run. The paper builds on SD-RPN, a lightweight proposal network that reuses the first frozen MLLM blocks, attaches a small stack of trainable blocks, and predicts a dense RoI logit map in a single pass from the last prefilling token. SD-RPN was trained by distilling response-to-image attention, a token-wise surrogate that may keep spurious activations or omit weakly attended evidence, and crucially is never checked against the answer.

The problem is that the RoI map reaches the answer only through a discrete region choice — the map is binarized, regions are extracted, the selected crop is re-encoded — so no gradient connects the answer back to the map, and fine-grained RoI annotations are not uniquely defined anyway.

Vision-RL² solves this by treating coherent regions (connected components of a smoothed, binarized map) as actions and using a frozen MLLM as a reader that scores each region's function. For a given mask, the reader is conditioned on the masked image and teacher-forced on the gold answer tokens; the geometric mean answer probability and its log-odds serve as a functional score. A region's contribution is its leave-one-out value: how much the score drops when that region is removed. This needs only n+1 reader evaluations for n regions, and needs no response sampling or reasoning trajectories.

Two marginal action groups use this signal. The subtractive group ranks the policy's own predicted regions by their mean RoI logit, keeps the top K, and compares each region's removal contribution against a calibrated noise margin estimated from two control regions grown outside the dilated prediction in low-probability areas. Regions contributing more than the margin are preserved; others have their confidence decreased. A softmax over the removals turns region confidence into a differentiable policy, so credit flows into the dense map even though region extraction itself is non-differentiable. The additive group tests supplementary regions taken from frozen response-to-image maps at six layers spread across depth, rewarding those whose inclusion raises the answer likelihood. Both policy losses are scaled by a detached attainability weight, anchored to the initial SD-RPN by a KL term, and supplemented by a binary cross-entropy term when only a single region is proposed. Only the predictor parameters are updated; the MLLM stays frozen.

At inference, sparse visual encoding measures the foreground occupancy of the predicted bounding box, computes a zoom factor as the square root of the inverse occupancy, caps it, and re-encodes the crop retaining only foreground tokens — the retained count approximately matches the dense crop budget while evidence is viewed at finer resolution. Position embeddings are assigned on the full bbox grid before background tokens are dropped, and source-image tokens are not re-encoded, with their KV cache reused in part of the LLM layers.

Training uses 7K QA pairs from the VisualCoT training corpus (5K InfographicVQA, 1K TextVQA, 1K DocVQA), selected from the top half of each 10K-candidate split by the standard deviation of region-removal rewards under the initial SD-RPN. The source image is encoded under a 576 visual-token limit during training; all models are trained for one epoch with batch size 32 and learning rate 1.5 × 10⁻⁵.

Why This Matters

Impact on research. The paper reframes fine-grained perception as two operations with distinct resolution needs, and supplies a quantitative measurement (roughly 3–4 times asymmetric compression tolerance, with the box surviving 25 times reduction and the crop failing at 9 times) that other work can build on. It also offers a general recipe — scoring candidate proposals by their measured effect on a frozen reader — for interfaces that decide where to spend computation, and shows that answer-level supervision with region-level credit can replace both geometric annotation and raw attention distillation.

Real-world applications:

• Document and infographic question answering, where answers depend on small text inside large, mostly irrelevant page areas; the training data deliberately draws on InfographicVQA, TextVQA, and DocVQA.
• High-resolution image inspection such as satellite or aerial imagery at HR-Bench 4K and 8K scales, where evidence occupies a small fraction of the frame.
• On-device or edge assistants where latency and compute budgets are tight, since the RPN head routes in 31–49 ms versus 1.1–1.8 s for attention routing and coordinate decoding.
• Cost-sensitive inference deployments, since the method matches SD-RPN accuracy with 4.2 times fewer visual tokens on the 4B model and responds 2.6 times faster than Vision-OPD-4B at its largest budget.

Industry relevance. The method updates only a small attached predictor rather than finetuning the full MLLM, training an order of magnitude fewer parameters than the full-finetuning baselines it compares against (DeepEyes, ZwZ, P2R, Vision-OPD). That makes it attractive for serving existing frozen backbones and for reusing a single trained localizer across scales, without the memory cost and instability the paper attributes to full-model reinforcement learning.

Future Directions

• Closing the MME-RealWorld gap. Full fine-tuning still wins on the MME-RealWorld EN and CN splits because it can adapt the reader itself; the paper explicitly forgoes this with a frozen reader, leaving open whether a lighter reader adaptation could recover those points without the full-finetuning cost.
• Extending beyond the predictor. The authors state they expect the recipe of scoring proposals by their measured effect on a frozen reader to extend to other interfaces that decide where to spend computation — an invitation to test it on routing problems beyond visual RoI selection.
• Richer action structures. Pair removals performed at parity with the full recipe at clear extra cost, and cell-level keep-sets lost 1.2 points. Whether more expressive but still coherent action units can beat leave-one-out under a better baseline remains open.
• Scaling and generalization of the diagnostic. The controlled intervention was run on ZoomBench with Qwen3.5-4B only; whether the 3–4 times localization-versus-recognition asymmetry holds at other model scales, and on benchmarks outside the six reported, is not established by the paper.

Target Audience

This paper is most useful to researchers and engineers working on efficient multimodal LLMs, high-resolution visual perception, and reinforcement learning applied to vision-language systems. It will particularly benefit readers interested in region proposal and routing interfaces, token-budget reduction for inference, and training methods that avoid region annotations or generated reasoning trajectories. Practitioners deploying MLLMs under latency or compute constraints will find the efficiency and routing-cost comparisons directly relevant, while readers new to MLLM internals may need background on visual tokenizers, attention maps, and policy-gradient terminology to follow the method section.

Authors’ abstract

Fine-grained visual perception in MLLMs is commonly improved by raising the resolution, but the added visual tokens inflate vision-encoding and language-model prefilling costs. We show that the two operations underlying fine-grained perception, localizing the region of interest (RoI) and recognizing its content, have different resolution requirements. In a controlled diagnostic, localization tolerates roughly 3 to 4 times stronger token compression than recognition, which motivates localizing from a coarse view and concentrating resolution on the selected evidence. Decoding coordinates with the MLLM can be trained end-to-end from answers, but costs a full model pass per query and depends on grounding ability. A lightweight proposal network distilled from the model's attention is fast, but inherits the noise of its attention targets. The RoI from the proposal network reaches the answer through a discrete region choice, so its faithfulness to the answer cannot supervise the network. We therefore optimize the proposal network with region-level reinforcement learning, which we call Vision-RL2. It treats coherent regions as actions, and a frozen MLLM reader scores each one by how its removal changes the answer likelihood. Complementary subtractive and additive objectives suppress distracting proposals and recover missing evidence, updating only the predictor without region annotations, response sampling, or reasoning trajectories. The refined proposal further enables a sparse encoding that magnifies evidence and excludes background tokens. Across six fine-grained benchmarks and four MLLM backbones, Vision-RL2 improves accuracy over the base model at every token budget and surpasses its largest-budget accuracy with about 4 times fewer visual tokens. Code is available at https://github.com/YuHengsss/VisionRL2 .