Visually-Guided Policy Optimization for Multimodal Reasoning

AI summary

Overview

Research area: Multimodal reasoning with vision-language models (VLMs), specifically reinforcement learning with verifiable rewards (RLVR) for improving visual grounding in chain-of-thought reasoning.

Technical level: Intermediate. The paper assumes familiarity with reinforcement learning for language models (GRPO/DAPO-style policy optimization), VLM architecture, and attention/hidden-state analysis, though its central ideas can be grasped without implementing the math.

One-sentence scope: The paper introduces Visually-Guided Policy Optimization (VGPO), a training framework that uses a model's own internal hidden states to measure and reinforce visual focus during reinforcement learning, countering both sparse visual attention and the gradual decay of visual attention over long reasoning chains.

What This Paper Is About

Vision-language models tend to reason mostly from text, paying little attention to the input image, and that attention fades further as reasoning chains get longer — a problem the authors call temporal visual forgetting. Existing fixes require extra special tokens, auxiliary models (for example, VAPO uses a GPT-5 model), or additional forward passes. VGPO instead re-weights the reinforcement learning advantage using only signals the model already produces internally, so visual attention is amplified and sustained without any external dependency.

Key Contributions

1. Visual Focus Score. The authors show that the hidden-state similarity between generated tokens and image tokens acts as a reliable, internal measure of visual grounding, allowing visual tokens to be localized with no external supervision.
2. Visual Attention Compensation (VAC). A mechanism that uses the visual focus score to amplify visual cues while progressively raising visual expectations in later decoding steps to counteract temporal visual forgetting.
3. Dual-Grained Advantage Re-Weighting. VGPO folds the compensation mechanism into policy optimization through both an intra-trajectory scaling factor (highlighting tokens with above-average visual activation) and an inter-trajectory scaling factor (favoring whole trajectories with greater cumulative visual focus).
4. Empirical validation. Experiments across mathematical and vision-dependent multimodal benchmarks show improved visual activation, reduced visual forgetting, and state-of-the-art results relative to comparable models.

Main Findings

• Text-dominated attention is measurable. In an analysis based on Qwen2.5-VL-7B, attention to generated text and the input query exceeds attention to image tokens, and visual activation appears as brief "glances" early in generation followed by long periods of neglect.
• Visual attention decays with reasoning length. Across four visual-dependent benchmarks, visual attention rises initially and then progressively falls as generation steps increase — the temporal visual forgetting phenomenon.
• Correct answers correlate with sustained visual attention. Correct samples had a higher late/early visual accumulation ratio than incorrect samples (Average, 0.680 vs. 0.532).
• Activated visual tokens are well-grounded. When visual tokens are activated, their attention maps on the original image are accurate and semantically grounded in current base models — the model can locate relevant information but does not sustain focus.
• Large gains on Qwen2.5-VL-7B. VGPO produced relative gains of 33.2% on general mathematical reasoning and 30.0% on vision-dependent reasoning over the base model, with average accuracies of 66.6% (Avg-Math) and 63.3% (Avg-Vision) — the best among the compared ~7B reasoning models.
• Competitive against much larger models. Despite the scale gap, the 7B model's results are described as comparable to Qwen2.5-VL-72B.
• Scales across model sizes. On Qwen2.5-VL-32B, VGPO reached 70.7 Avg-Math (a 13.8% relative gain) and 66.7 Avg-Vision (10.8% relative gain) over DAPO; on Qwen2.5-VL-3B it reached 57.7 Avg-Math (30.8%) and 53.6 Avg-Vision (30.7%).
• Works with less data. With Geo3K (2.1K), VGPO reached 60.4 Avg-Math (20.8%) and 55.8 Avg-Vision (14.6%); with MMK12 (6.4K), 62.4 Avg-Math (24.8%) and 60.3 Avg-Vision (23.8%) — surpassing the DAPO baseline in data-constrained regimes.
• Both re-weighting levels help. Ablation on Qwen2.5-VL-7B: DAPO baseline 63.8 Avg-Math / 59.6 Avg-Vision / 62.2 Overall; adding intra-trajectory 66.1 / 62.5 / 64.6; adding inter-trajectory 65.3 / 62.0 / 64.0; combining both 66.6 / 63.3 / 65.3.
• Beats generic advantage shaping. Entropy-based shaping achieved 65.6 / 61.9 / 64.1 and KL-based shaping 65.7 / 61.3 / 63.9, both below VGPO's 66.6 / 63.3 / 65.3.
• Linear compensation beats other schedules. Linear (66.6 / 63.3 / 65.3) outperformed Exponential (65.1 / 61.0 / 63.5) and Step-Function (64.7 / 60.7 / 63.1); exponential over-corrects on final tokens and step-function introduces instability.
• Late-stage compensation is essential. Full-trajectory compensation collapsed performance to 53.0 / 54.2 / 53.5, versus 66.6 / 63.3 / 65.3 for late-trajectory compensation.
• Hyperparameter equilibrium matters. Best results occurred at β = 0.3, κ = 0.2, and γ = 0.5, and the model performed best when the Late/Early Ratio converged to or stabilized near 1.
• Sustained attention after training. Comparing visual attention ratio before and after VGPO across four vision-dependent benchmarks showed higher attention throughout generation and better mitigation of temporal forgetting.

Methodology in Plain English

The approach builds on Group Relative Policy Optimization (GRPO), which trains a model by sampling a group of candidate answers, scoring each as right or wrong, and normalizing those scores within the group to get an advantage. Following DAPO, the authors use clip-higher asymmetric clipping and no KL penalty.

On top of that standard setup, VGPO adds three pieces:

1. Measure visual focus internally. The hidden states of all input image tokens are pooled into a single "visual prototype." For each generated token, the cosine similarity between its hidden state and that prototype is computed and rescaled to the range [0, 1] — this is the Visual Focus Score. No extra model, token, or forward pass is needed.

2. Compensate for decay. Because focus scores naturally sink in later reasoning steps, the raw score is multiplied by a factor that grows linearly with the relative position of the token in the trajectory. A gate switches this on only for tokens in the tail of the trajectory (past a fraction γ of the length) whose score is in the top κ-percent of that tail, filtering out noisy text tokens. β controls how strong the compensation is.

3. Reshape the advantage. The per-token score is min-max normalized and zero-centered within each trajectory to produce an intra-trajectory factor, so only above-average visual tokens get boosted. Separately, the total accumulated score over a whole trajectory is normalized and zero-centered across the rollout group to produce an inter-trajectory factor. The final advantage multiplies the base outcome advantage by (1 + intra factor) × (1 + inter factor), so the policy is rewarded not just for correct answers but for visually faithful reasoning paths.

The models used are Qwen2.5-VL at 3B, 7B, and 32B. Training data is ViRL39K (roughly 39,000 samples), Geo3K, and MMK12, with validation on MMK12-val. Training uses 2 epochs, a learning rate of 1×10⁻⁶, rollout batch size of 512, and maximum response length of 2,048; evaluation uses temperature 0.0. Evaluation benchmarks are drawn from PAPO-Eval, covering MathVista, MathVerse, We-Math, MMK12, GeoMath, Geometry3K, LogicVista, SuperClevr Counting, MMMU-Pro, and MathVerse-V.

Why This Matters

Impact on research. The work reframes multimodal RLVR: rather than adding external supervision (extra tokens, auxiliary verifiers, or dual forward passes as in Look-Back, VAPO, PAPO, and VPPO), it shows that a model's own hidden states carry enough signal to locate and reinforce visual grounding. The empirical characterization of temporal visual forgetting — and its correlation with correctness — gives the field a concrete diagnostic to build on.

Real-world applications:

• Chart, table, and document question answering, where answers must be grounded in specific visual elements rather than textual priors.
• Geometry and STEM problem solving from diagrams, one of the benchmark families used (GeoMath, Geometry3K, MathVerse).
• Visual counting and spatial reasoning tasks, as covered by SuperClevr Counting and LogicVista.
• General multimodal assistants used in high-stakes settings, where hallucinated visual claims are costly.

Industry relevance. The method adds no auxiliary model, no extra tokens, and no additional forward passes, which matters for training cost at scale. It also proves portable across 3B, 7B, and 32B backbones and across datasets of different sizes (39K, 6.4K, 2.1K), making it attractive to teams that cannot afford large-scale data collection. The code is released at https://github.com/wzb-bupt/VGPO.

Future Directions

1. Adaptive rather than fixed compensation. The authors note that progressively elevating visual expectations is a heuristic that may not be optimal when final reasoning steps are purely logical or computational; an adaptive mechanism that adjusts visual reliance based on context is left open.
2. Overcoming the visual encoder ceiling. VGPO's upper bound is constrained by the representational quality of the visual encoder and projector — if critical visual features are never encoded into the hidden states, re-weighting cannot recover them. Improving raw perception is separate future work.
3. Confirmation across other architectures and modalities. The paper tests Qwen2.5-VL at three scales; whether the Visual Focus Score behaves comparably in other VLM families or in video and other visual inputs is not reported.
4. Extending beyond answers. Whether reward strategies can be built directly on visual grounding for tasks without verifiable final answers (where RLVR's exact-match reward is unavailable) is not addressed by this work.

Target Audience

Researchers and engineers working on reinforcement learning for vision-language models, multimodal reasoning, and visual grounding. It is most useful to those already familiar with GRPO/DAPO-style policy optimization who want a lightweight, self-supervised way to improve visual faithfulness, and to practitioners who need strong 7B-scale multimodal reasoning without auxiliary models or large training sets. Readers new to RL for language models will need background reading first, since the methodology section is written for an informed audience.

Authors’ abstract

Reinforcement learning with verifiable rewards (RLVR) has significantly advanced the reasoning ability of vision-language models (VLMs). However, the inherent text-dominated nature of VLMs often leads to insufficient visual faithfulness, characterized by sparse attention activation to visual tokens. More importantly, our empirical analysis reveals that temporal visual forgetting along reasoning steps exacerbates this deficiency. To bridge this gap, we propose Visually-Guided Policy Optimization (VGPO), a novel framework to reinforce visual focus during policy optimization. Specifically, VGPO initially introduces a Visual Attention Compensation mechanism that leverages visual similarity to localize and amplify visual cues, while progressively elevating visual expectations in later steps to counteract visual forgetting. Building on this mechanism, we implement a dual-grained advantage re-weighting strategy: the intra-trajectory level highlights tokens exhibiting relatively high visual activation, while the inter-trajectory level prioritizes trajectories demonstrating superior visual accumulation. Extensive experiments demonstrate that VGPO achieves better visual activation and superior performance in mathematical multimodal reasoning and visual-dependent tasks.