Turns Matter: Advancing Streaming Video RL via Turn-Level Credit

AI summary

Overview

Research area: Computer Vision, specifically reinforcement learning for streaming (online) video-language models that must decide turn-by-turn whether to speak or stay silent.

Technical level: Advanced — the paper assumes familiarity with Group Relative Policy Optimization (GRPO), policy-gradient surrogates, advantage normalization, and credit assignment, though the core intuition is explained in accessible terms.

Scope: The paper proposes TurnWise, a turn-level GRPO framework that replaces a single shared trajectory advantage with per-turn remaining-return advantages and reweights the policy loss so every response-or-wait decision counts equally, evaluated on three streaming video benchmarks with two Qwen backbones.

What This Paper Is About

Streaming video assistants watch video arrive over time and must repeatedly decide, at each decision point, whether to answer the user or keep observing silently. Recent work applies GRPO to this setting but still assigns one trajectory-level advantage to every turn in a dialogue, so a redundant reply after an earlier success inherits positive credit, and a genuinely useful follow-up after earlier mistakes inherits negative credit. TurnWise fixes this mismatch by scoring each turn's remaining return and by weighting each decision equally in the loss, regardless of how long its text output is.

Key Contributions

1. Identifies a granularity mismatch in streaming video RL. The authors show that shared trajectory advantages obscure the value of individual later decisions, while token-mean loss aggregation makes each turn's weight depend on its response length rather than its importance.

2. Introduces silence-completed prefix scoring for turn-specific credit. By re-scoring each trajectory prefix with all remaining turns forced to the waiting action (NO REPLY), the method converts trajectory feedback into group-relative remaining-return advantages without a learned critic and without extra rollouts or judge calls.

3. Introduces turn-mean loss aggregation. Token losses are averaged within each turn and then across turns, so waiting actions and long responses receive the same weight 1/T_i in the policy loss.

4. Demonstrates gains on three streaming benchmarks with two backbones, including a component analysis showing that turn-mean aggregation improves response timing and reduces repetition relative to credit-only optimization.

Main Findings

• Best aggregate scores across all three benchmarks with both backbones. On Qwen2.5-VL-3B, TurnWise reaches 43.56 PAUC on ProactiveVQA, 21.59 average Timing F1 on OVO-Timing, and 49.43 average accuracy on StreamingBench, exceeding the strongest non-TurnWise results by 3.96, 3.87, and 0.69 points respectively. On Qwen3-VL-4B it reaches 42.44, 11.15, and 58.96, with gains of 6.26, 1.92, and 2.38 points.

• ProactiveVQA category leadership. TurnWise leads all four categories (Ego, TV, VAD, Web) with the 3B backbone and three of four with the 4B backbone; for the 4B model, VAD and Web improve by 3.49 and 6.43 points over the strongest competing results.

• StreamingBench improvements with the 4B model. Proactive accuracy reaches 38.40 versus 35.60 for the strongest baseline, while real-time accuracy rises from 78.24 to 79.52.

• Credit assignment and turn-mean aggregation are complementary. On Qwen2.5-VL-3B, combining both reduces repetition by 61.5% relative to credit-only optimization (15.53 to 5.98) while raising Timing F1 by 2.96 points and StreamingBench average accuracy by 0.49 points. Relative to turn-mean aggregation alone, the combination improves Timing F1 by 13.42 points and reduces repetition by 38.9%.

• Full credit correction is the best operating point. Within the complete framework, α = 1 achieves the highest PAUC (43.56), Timing F1 (21.59), and StreamingBench average (49.43) and the lowest repetition (5.98) among the tested correction strengths of 0.25, 0.50, 0.75, and 1.00.

• Behavior shifts toward more selective responding. Turn-mean aggregation raises mean reward from 0.657 to 0.711 over the final 50 updates while reducing repetition-penalty magnitude from 0.069 to 0.036 across 497 updates; the outside-window reply fraction falls from 0.537 to 0.289 and the non-silent decision fraction from 62.2% to 37.0%.

• Advantages are genuinely redistributed. Turn-level credit assigns negative advantages to 59.83% of repetitive replies that receive positive global advantages (432 of 722). Over 11,116 complete Qwen2.5-VL-3B training trajectories and 84,184 decision turns, 21.20% of decisions change advantage sign relative to global allocation and 53.15% of trajectories receive both positive and negative turn advantages. Across all 2,972 Judge-labeled repetitive replies, the share receiving negative advantages rises from 75.7% to 81.6%.

Methodology in Plain English

The setup: a model watches a video stream and, at a fixed grid of decision points, either writes a reply or outputs the explicit waiting action NO REPLY. A group of G = 4 trajectories is sampled per instance, and each complete trajectory is scored by a fixed PAUC-based reward: 3P_i − 2D_i^rep − 0.5D_i^out − 2D_i^prefix, covering answer quality and timeliness, repetition, out-of-window replies, and overly long shared prefixes. Rewards are judged by gpt-oss-20b.

Standard GRPO turns each trajectory's score into one number — how far above or below the group average it is — and applies that same number to every token of every turn. TurnWise instead asks: what did this turn actually add? It answers this by truncating the trajectory at turn t, forcing every later turn to silence, and re-scoring. That prefix score captures reward already banked; subtracting the group's average banked reward from the trajectory's full reward leaves a "return to go" that starts fresh at that decision. A scalar α controls how much of the earlier reward is removed, warmed up linearly from 0 to 1 over the first 10% of updates, and advantages are clipped to ±5 using the same group standard deviation as ordinary GRPO, so a first turn behaves exactly like standard GRPO.

The second piece addresses weighting. Under token-mean aggregation, a long reply gets more of the loss than a short silence, which biases the model toward verbose behavior. TurnWise averages the token losses inside each turn, then averages across turns, so each response-or-wait decision occupies 1/T_i of the trajectory loss.

Training details: SFT on 100,874 instruction examples (1% held out) from EgoExo4D, EgoExoLearn, Live-WhisperX, Tarsier2, and LLaVA-Video for one epoch at learning rate 10⁻⁵ with the vision tower frozen; RL on 3,983 streaming instances with 20-second target segments and at most 90 frames sampled every two seconds; batch size 8, temperature 1.2, top-k = 10; one epoch with AdamW at learning rate 10⁻⁶, no KL or entropy regularization; ε = 10⁻⁶, δ = 0.2, c = 3.

Why This Matters

Impact on research. The paper reframes a problem the streaming-video RL literature has treated as a reward-design problem as instead a credit-assignment and loss-weighting problem. Because the correction reuses cached reward evaluations and needs no critic model or extra rollouts, it is a drop-in change to existing GRPO pipelines, and the decomposition is shown to be exact even for non-additive, ratio-based reward terms like the repetition fraction.

Real-world applications:

• Always-on video assistants for smart glasses or phone cameras that must decide when to speak up without narrating constantly.
• Security and monitoring systems that should alert on a genuine event within a time window and stay quiet otherwise.
• Live-stream moderation or commentary where redundant repeated flags and missed late-breaking events both carry cost.
• Assistive technologies for accessibility, where timely description matters and repetitive narration is actively disruptive.

Industry relevance. The method adds no learned critic and no additional rollouts or judge calls, so the compute overhead is a deterministic scan over already-computed scores — attractive for teams already running GRPO-style RL on video models. The reported behavior change (non-silent decisions dropping from 62.2% to 37.0% while reward rises) maps directly onto production concerns about chatty assistants and API/judge cost.

Future Directions

• Extending beyond the PAUC-based reward. All experiments use the MMDuet2-style scorer judged by gpt-oss-20b; whether the remaining-return decomposition transfers to other streaming reward formulations or learned reward models is untested here.
• Broadening the backbone and benchmark range. Results cover Qwen2.5-VL-3B and Qwen3-VL-4B on ProactiveVQA (300 instances), OVO-Timing, and StreamingBench; larger models and other streaming benchmarks are not reported.
• Understanding the α schedule. Full correction (α = 1) wins within the complete framework, but the credit-only column in the sensitivity table is non-monotonic (peak PAUC of 44.50 at α = 0.25), leaving open how best to schedule or adapt the warm-up.
• Longer-horizon credit. Prefix rescoring cost grows with the number of decision turns; the paper does not report how the approach scales to substantially longer streams than the 20-second, at-most-90-frame segments used in training.

Target Audience

Researchers and engineers working on streaming or online video-language models, and anyone applying GRPO-style reinforcement learning to multi-turn or agentic settings where a single trajectory-level reward must be split across many decisions. It is most useful to readers already comfortable with policy-gradient methods; the two design ideas themselves are describable without heavy math, but the derivations and experimental comparisons assume an RL background.

Authors’ abstract

Streaming video understanding requires models to repeatedly decide whether to respond or continue observing. Recently, Group Relative Policy Optimization (GRPO) has been applied to this setting, with existing approaches emphasizing reward design while retaining a single trajectory-level advantage across turns. Such shared credit can reinforce redundant replies after earlier successes and discourage useful follow-ups after earlier failures. We propose TurnWise, a turn-level GRPO framework that aligns learning with individual streaming decisions. TurnWise addresses the mismatch between trajectory-level feedback and turn-level interaction, promoting informative responses and reducing redundancy. Specifically, turn-level credit assignment uses silence-completed prefix scoring to construct group-relative advantages from each decision's remaining return, separating its learning signal from earlier rewards. Turn-mean policy optimization averages token losses within each turn and then across turns, giving response and waiting decisions equal weight regardless of output length. These complementary designs improve streaming policy learning without a learned critic or additional rollouts. Extensive experiments show our method outperforms prior RL approaches for streaming video understanding.