Video-KTR: Reinforcing Video Reasoning via Key Token Attribution

AI summary

Overview

Research area: Computer vision and multimodal large language models (MLLMs), specifically reinforcement learning (RL) for video reasoning.

Technical level: Intermediate. The paper assumes familiarity with reinforcement learning for language models (GRPO, policy gradients, advantage estimates) and with vision-language model training, but its central ideas—selectively updating only the most informative output tokens—are explained in accessible terms.

Scope: This paper introduces Video-KTR, a token-level reinforcement learning framework that identifies and reinforces only the output tokens most sensitive to visual input, temporal ordering, and predictive uncertainty, and evaluates it across five video benchmarks.

What This Paper Is About

Most reinforcement learning methods for video reasoning assign rewards at the level of the whole answer sequence, or select tokens using only a single signal such as model uncertainty (entropy). This is coarse: it treats a visually descriptive noun, a temporal connective like "then," and a filler word like "the" as equally important, and it obscures which parts of an answer actually depend on the video versus on language priors. Video-KTR's goal is to identify the small set of output tokens that genuinely depend on visual content or temporal order, and to update only those tokens during RL training, improving both accuracy and interpretability.

Key Contributions

1. Modality-aware token attribution via counterfactual analysis. The authors propose three complementary attribution signals for identifying critical tokens: visual-aware tokens found by masking the video and measuring token-level log-probability shifts; temporal-aware tokens found by shuffling video frames and measuring the resulting shifts; and high-entropy tokens that mark predictive uncertainty. The paper states this is the first work to integrate modality-aware token selection into RL for video reasoning.

2. Video-KTR, a token-level policy shaping framework. Rather than applying a coarse sequence-level reward, the method takes the union of the top r% of tokens from each attribution strategy and applies a binary mask so that only those key tokens contribute to the GRPO-style policy update.

3. State-of-the-art or competitive results across five benchmarks. Video-KTR reaches 42.7% on Video-Holmes (compared with GPT-4o's 42.0%), 53.1% on VideoMMMU, 66.6% on MMVU(mc), 73.5% on TempCompass, and 62.5% on VideoMME at 64 input frames, with accuracy rising steadily as frames increase from 16 to 64.

4. Systematic ablations of signals, weighting schemes, update ratios, and perturbations. The paper isolates the contribution of each attribution signal, compares hard versus soft token weighting, sweeps the token update ratio, and tests alternative perturbation strategies, along with a gradient-level analysis of why selective updating works.

Main Findings

• Video-Holmes performance. Video-KTR reaches 42.7% overall on Video-Holmes, versus GPT-4o's 42.0%. For reference, the table lists GPT-5 at 46.7 (evaluated by the authors), Gemini-2.5-Pro at 45.0, and Gemini-1.5-Pro at 41.3. Among open-source 7B–8B baselines, Video-R1 is listed at 36.5, TW-GRPO at 32.9, and Qwen2.5-VL at 27.8.

• Consistent gains across benchmarks and frame counts. At 16 frames Video-KTR scores 40.7 / 51.3 / 65.7 / 73.3 / 57.3 on Video-Holmes / VideoMMMU / MMVU / TempCompass / VideoMME; at 32 frames 41.6 / 52.6 / 65.9 / 73.4 / 60.3; and at 64 frames 42.7 / 53.1 / 66.6 / 73.5 / 62.5.

• All three attribution signals are complementary. Every individual signal beats vanilla GRPO (38.8 Video-Holmes, 49.8 VideoMMMU, 64.8 MMVU, 51.1 average). Temporal-aware alone gives the largest Video-Holmes gain (42.1) but hurts other benchmarks. The full combination E+V+T achieves the best average (53.4) and the best VideoMMMU (52.6) and MMVU (65.9) among ablations.

• Selected tokens differ linguistically by strategy. Visual-aware tokens are dominated by nouns (24.8%), temporal-aware tokens by verbs (21.2%) and pronouns (11.0%), and entropy-aware tokens by adverbs (8.8%). The selected tokens are content words such as "object," "person," "appear," "hold," "wait," "finally," while unselected tokens are function words—auxiliaries, pronouns, determiners, prepositions, and punctuation.

• Hard top-20% selection beats soft weighting. Binary Top-20% selection outperforms softmax, sigmoid, linear, and exponential weighting variants. The union of the three signals produces a final update ratio of roughly 40%, with most tokens unique to a single strategy.

• Log-probability difference is the most reliable divergence measure. Simple log-probability differences between pre- and post-perturbation predictions outperform L1/L2 norms, KL/JS divergences, cosine similarity, and Hellinger distance, and require only one subtraction per token.

• A 20% update ratio is optimal. Accuracy follows a double-peak pattern with a best result at a 20% ratio; performance drops at 30–50%, indicating that overly large update sets introduce noise.

• Perturbation choice matters little. Full-frame masking (52.5) and random shuffling (52.6) slightly edge out masking half, replacing frames with unrelated content, sequence reversal, and segmental shuffling, suggesting limited sensitivity to perturbation strength.

• Gradient analysis explains the gains. The mean gradient norm of selected tokens is 4.50, roughly three times higher than that of masked tokens. Cosine similarity with the full gradient averages 0.862 for critical tokens versus 0.572 for masked tokens. Training loss variance is 0.4436 for Video-KTR versus 0.5528 for vanilla GRPO.

• Gains extend to image benchmarks. Video-KTR records 47.33 on MMMU(val), 80.34 on AI2D, 57.77 on MMStar, and 80.68 on ChartQA. A visual-tokens-only variant already beats GRPO baselines, but the full model is best everywhere.

• Marginal computational overhead. Training used 32 H100 (80GB) GPUs and completed in 5.2 hours. Peak memory rises from 77.9 GB (vanilla GRPO) to 78.5 GB, forward latency from 4.70 s to 4.92 s, and forward FLOPs per GPU from 116.6 T to 129.9 T. Samples per second fall slightly from 0.946 to 0.855.

• The approach generalizes to a smaller backbone. On Qwen2.5-VL-3B, Video-KTR beats vanilla GRPO at 16, 32, and 64 frames across Video-Holmes, VideoMMMU, MMVU, TempCompass, and VideoMME.

• Updates concentrate early but are not confined there. Earlier token positions receive higher update probabilities, while later positions still receive non-negligible updates.

Methodology in Plain English

The authors start from an existing supervised-fine-tuned video model (Video-R1-SFT built on Qwen-2.5-VL-7B) and apply reinforcement learning on top of it.

First, they generate an answer to a video question. Then, for each token in that answer, they ask three separate questions:

1. Does this token depend on seeing the video? They re-run the model with the video inputs blanked out and measure how much the token's log-probability changes. Big change means the token is visually grounded.
2. Does this token depend on the order of events? They shuffle the video frames and measure the log-probability change again. Big change means the token is temporally sensitive.
3. Is the model unsure about this token? They compute the entropy of the model's prediction distribution. High entropy marks decision points and uncertain reasoning.

They rank tokens by each signal, take the top r% from each, and combine those sets into one "key token" set. During the GRPO policy update, a binary mask zeroes out the loss contribution of all tokens outside this set, so only the key tokens drive learning.

For training data, they use the 260K samples released by Video-R1 and filter out questions that the SFT model answers above 80% or below 20% of the time (over 8 rollouts), leaving roughly 15K samples, then add 1.5K Video-Holmes training examples. Training limits inputs to 16 frames at 128×28×28 resolution, uses a learning rate of 2e-6, a global batch size of 32, a rollout batch size of 256, a maximum sequence length of 16384, a KL coefficient of 0.4, 8 sampled responses per prompt, and temperature 1.0. At inference, frames can extend to 64 at 256×28×28 resolution.

They then run ablations: removing each attribution signal, replacing hard selection with soft weighting schemes, varying the proportion of tokens updated, and swapping in different perturbation strategies.

Why This Matters

Impact on research. The paper argues that coarse sequence-level rewards and single-signal token selection leave the link between visual input, temporal structure, and generated language underspecified. By showing that attributing tokens to specific modalities improves both accuracy and interpretability, it offers a drop-in modification to existing RL pipelines for video models, and it demonstrates that performance gains need not come at the cost of general video understanding. The gradient analysis provides a mechanistic account of why masking low-information tokens stabilizes training rather than simply reporting benchmark deltas.

Real-world applications.

• Video question answering and assistants that must reason about event order, such as "what did the person do after entering the room?"
• Surveillance or safety monitoring, where causal and temporal relationships between events carry the important signal.
• Educational and instructional video tools that need to track sequential procedures and explain them.
• Content analysis of narrative media such as short films, the setting used by the Video-Holmes benchmark, including timeline reconstruction and theme inference.

Industry relevance. The method integrates into existing GRPO-based training code with one extra forward pass, adds little memory or latency overhead, and trains in 5.2 hours on 32 H100 GPUs—making it a practical option for teams already running RL post-training on video-language models. The robustness to smaller backbones (Qwen2.5-VL-3B) also suggests the recipe is not limited to large compute budgets. Code and models are released at https://github.com/zywang0104/Video-KTR.

Future Directions

• Robustness under degraded visual conditions. The authors note that low light, occlusion, rapid motion, and OCR/ASR noise may disrupt visual-text alignment, and flag stabilizing key token selection in these settings as open work.
• Extension beyond question answering. Video captioning and temporal grounding were not systematically examined, so the framework's generalization to these tasks remains untested.
• Incorporating additional modalities. The current framework covers vision and language only; audio and motion sensor data are identified as promising complementary signals.
• Interaction between update ratio and signal mix. Since 20% was optimal with all signals combined and results degraded at 30–50%, how the ideal ratio shifts with different signal combinations or model sizes is not reported.

Target Audience

Researchers and engineers working on multimodal large language models, reinforcement learning post-training, and video understanding. It will be most useful to practitioners who already know GRPO-style RL and want a concrete, low-overhead way to make reward assignment more fine-grained and modality-aware, as well as to readers interested in interpretability of vision-language reasoning. Readers without background in RL for language models will need to consult the cited GRPO work first.

Authors’ abstract

Reinforcement learning (RL) has shown strong potential for enhancing reasoning in multimodal large language models, yet existing video reasoning methods often rely on coarse sequence-level rewards or single-factor token selection, neglecting fine-grained links among visual inputs, temporal dynamics, and linguistic outputs, limiting both accuracy and interpretability. We propose Video-KTR, a modality-aware policy shaping framework that performs selective, token-level RL by combining three attribution signals: (1) visual-aware tokens identified via counterfactual masking to reveal perceptual dependence; (2) temporal-aware tokens detected through frame shuffling to expose temporal sensitivity; and (3) high-entropy tokens signaling predictive uncertainty. By reinforcing only these key tokens, Video-KTR focuses learning on semantically informative, modality-sensitive content while filtering out low-value tokens. Across five challenging benchmarks, Video-KTR achieves state-of-the-art or highly competitive results, achieving 42.7\% on Video-Holmes (surpassing GPT-4o) with consistent gains on both reasoning and general video understanding tasks. Ablation studies verify the complementary roles of the attribution signals and the robustness of targeted token-level updates. Overall, Video-KTR improves accuracy and interpretability, offering a simple, drop-in extension to RL for complex video reasoning. Our code and models are available at https://github.com/zywang0104/Video-KTR.