Reviso: Not All Visual Evidence Deserves Equal Attention in Video Captioning

AI summary

Overview

Research area: Computer Vision / multimodal large language models — specifically agentic (tool-using) video captioning and video understanding.

Technical level: Advanced. The paper assumes familiarity with multimodal LLMs, agentic tool-use loops, supervised fine-tuning, and reinforcement learning with GRPO-style group-relative rewards.

Scope: Reviso is an agentic video captioning framework built on Qwen3.5-9B that builds a hierarchical Context–Focus–Points observation plan from a low-frame-rate preview, then uses tools to gather targeted evidence for the information it judges most important, trained with verified trajectories and a three-part GRPO reward.

What This Paper Is About

Modern multimodal models can produce very detailed video captions, but detail is not the same as relevance: they often dwell on minor visual details while missing the key events and context that make a video understandable. The authors argue that visual evidence should not be treated equally, since what counts as important depends on each video's content — action order in a tutorial, a character's expression in a film scene, or a product's name and attributes in an advertisement. Reviso's goal is to learn what to look at in a given video and how to go back and look at it closely, rather than captioning every frame the same way.

Key Contributions

1. A priority-aware framing of video captioning. The paper argues that not all visual evidence deserves equal attention, and reframes captioning as identifying video-specific observation priorities and actively gathering evidence, rather than treating all visual content uniformly.

2. The Reviso framework. An agentic captioner that builds a hierarchical Context–Focus–Points (CFP) plan from a low-frame-rate global preview, then performs goal-directed active perception — selectively revisiting temporal segments at higher frame rates and combining tools such as image cropping and OCR — updating its plan as targets get resolved.

3. A verification-guided trajectory pipeline plus tailored GRPO rewards. Training trajectories are built in six steps: generate a plan and trajectory with Qwen3.8-27B, decompose ground-truth and generated captions into atomic facts, verify with Gemini 3.1 Pro, retry up to N times, fall back to teacher-guided recovery with verifier feedback, and retain only verified trajectories. GRPO then uses a Key-Fact Reward (KFR) for caption quality, a Plan Alignment Reward (PAR) for identifying important observation targets, and an Unproductive Tool-Call Penalty (UTCP) that scales the cost penalty with caption quality.

4. Empirical validation on four captioning benchmarks plus two downstream QA benchmarks, showing gains over the backbone and over the evaluated open-source baselines, and captions that better support downstream video question answering.

Main Findings

• DREAM-1K: Reviso reaches 42.3 precision, 37.0 recall, and 39.5 F1, versus 36.3 / 34.0 / 35.1 for its Qwen3.5-9B backbone — a 4.4-point F1 gain and the highest precision in the table.
• MCTS-VCB: The largest improvements appear here: precision 65.7 (+8.1), recall 56.8 (+19.1), F1 60.9 (+15.4) over the backbone's 57.6 / 37.7 / 45.5.
• CaReBench: Reviso obtains the highest action and object F1 among evaluated open-source models, at 34.3 and 33.9, versus 25.2 and 27.2 for the backbone (+9.1 and +6.7).
• VDC Detailed: Reviso reaches a TP score of 2.95 and TP accuracy of 0.568, the best among evaluated open-source models (GPT-6 Astra reports 3.00 and 0.576 but is excluded from ranking as proprietary).
• Captions transfer to downstream QA: Using Gemini 3.1 Pro to answer benchmark questions from generated captions alone (captions produced without seeing the questions), Reviso reaches 52.0 on LVBench (+4.9 over the backbone's 47.1) and 72.7 MMVU Overall (+2.1 over 70.6), with 81.7 MCQ and 57.9 open-ended.
• Every training stage helps: Base 35.1 / 25.2 / 27.2 → SFT only 37.6 / 30.7 / 30.1 → full model 39.5 / 34.3 / 33.9 on DREAM-1K F1, CaReBench action F1, and object F1 respectively.
• Tool use matters: A variant without active tool-based captioning stays near the base model (35.3 / 24.8 / 27.4), while removing the whole active-observation loop costs the most in the ablations.
• PAR removal hurts most among reward ablations: Dropping PAR yields 37.9 / 31.3 / 29.7, including a 4.2-point drop in object F1; dropping KFR gives 38.2 / 32.9 / 32.1 and dropping UTCP gives 38.4 / 33.7 / 32.4, versus 39.5 / 34.3 / 33.9 for the full objective.
• Targeted observation pays off most on longer videos: On CaReBench, tool-assisted observation improves object F1 by roughly 10 percentage points for clips lasting 24–27 seconds, with limited gains on shorter clips where the frame budget already gives dense coverage.
• More frames is not automatically better: Under direct captioning at 2, 4, 6, and 10 FPS on DREAM-1K, Molmo2-8B and Qwen3.8-27B keep improving with denser sampling, while Qwen3.5-9B and GPT-6 Astra decline at 10 FPS — and Reviso still beats all evaluated direct-captioning configurations (its F1 of 39.5 is marked as the reference line).
• Tools alone are not enough: In comparisons against Codex and Claude Code agentic workflows on five VLMs, Reviso outperforms most model–harness combinations; only GPT-6 Astra paired with Codex achieves a higher F1.
• Qualitative example: On a basketball clip, the backbone misreads a gesture as celebrating victory; Reviso re-inspects the shot, uses cropping and OCR on the scoreboard, and correctly reports Denver leading 74–72 with 1:45 left in the fourth quarter.

Methodology in Plain English

Reviso works in two phases, mirroring how a person might study a video before describing it.

First, get the gist cheaply. The model samples frames at a low frame rate to build a global preview — a cheap overview rather than the whole video at full density. From that preview it writes a three-level plan: the Context (what kind of video this is, the setting and main activities), the Focus (which aspects matter most for understanding it), and the Points (concrete things to check, together with which tool to use — a high-frame-rate revisit of a time segment, a crop, or OCR).

Second, go look. Following the plan, the model runs several rounds of tool-assisted inspection, interpreting what it sees alongside the global context, deciding which targets are now resolved, and revising the plan to chase what is still unclear. It then merges global context with the local evidence into the final caption.

How it learns this behavior. The authors could not rely on ordinary caption data, because they needed examples of the whole process — plans, tool calls, interpretations, revisions. They built a six-step pipeline: gather videos and captions from public datasets, generate extra candidate captions with GPT-6 Astra and Gemini 3.1 Pro, and have human annotators select, merge, correct, and supplement them into ground truth. Qwen3.8-27B then produces a full trajectory, which is checked by breaking both ground-truth and generated captions into atomic facts and having Gemini 3.1 Pro test factual consistency, reference-fact coverage, and whether the collected evidence actually supports the claims. Failed attempts are retried up to N times; persistent failures go to a teacher-guided recovery stage where Gemini 3.1 Pro produces a trajectory, receives verifier feedback about missing or unsupported facts, and revises. Only verified trajectories are kept.

Training. Reviso starts from Qwen3.5-9B and is fine-tuned on 10,000 verified trajectories for two LoRA epochs, with environment responses masked out of the loss. It is then trained with 200 GRPO steps on 20,000 samples, with the visual encoder frozen. The reward combines KFR (F1 between generated and reference fact sets), PAR (F1 between observation targets in the initial plan and the reference targets that a global preview cannot resolve), and UTCP (a cost penalty that only activates when at least one trajectory in the group clears a KFR threshold, and which is larger for lower-quality captions at equal cost and for costlier observations at equal quality). All models are capped at 128 input frames; baselines sample uniformly, while Reviso splits the same budget between preview and revisits.

Why This Matters

Impact on research. The paper moves video captioning away from "describe everything the encoder saw" toward learned, content-dependent evidence gathering. It also contributes a reusable recipe — verified agentic trajectories plus reward shaping that separates what to look at (PAR) from how much it cost (UTCP) — that applies beyond captioning to any agentic perception task. The finding that higher FPS can hurt strong models is a direct challenge to the common practice of scaling frame counts.

Real-world applications:

• Accessibility and description services — captions that prioritize plot-critical actions and context rather than incidental background detail.
• Sports and broadcast analysis — reading scoreboards, clocks, and game state via OCR and cropping, as in the paper's case study.
• Instructional and tutorial content — preserving action order and step-level correctness when summarizing how-to videos.
• Commerce and advertising — reliably capturing product names and attributes, which the paper identifies as the priority for ad content.
• Surveillance, robotics, and egocentric assistants — selectively revisiting moments instead of processing every frame at full density, which matters where compute and bandwidth are constrained.

Industry relevance. The frame-budget result (a fixed 128-frame cap split between preview and revisits) and the UTCP cost penalty both target deployment economics: teams can spend a bounded observation budget and still get better captions. The verification pipeline is also an operational template for turning expensive proprietary model outputs into trainable supervision for a much smaller open model.

Future Directions

• How should the observation budget be allocated? The paper shows gains concentrate in longer clips (~10 points for 24–27 s videos) but does not settle how preview-versus-revisit frame splits should be chosen per video.
• How far does the approach scale? Reviso is built on Qwen3.5-9B with a frozen visual encoder and 200 GRPO steps; whether the observation policy transfers to larger backbones or longer-form video is untested here.
• Can the verifier be less dependent on proprietary models? Ground-truth construction, verification, and teacher recovery all rely on Gemini 3.1 Pro, and candidate captions on GPT-6 Astra and Gemini 3.1 Pro — a dependency worth removing.
• What are the right reward components and weights? The ablation shows PAR matters most, but the weights α, β, and μ and the KFR threshold τ_q are not analyzed, leaving open how sensitive the policy is to this shaping.
• Where does agentic inspection still lose? GPT-6 Astra with Codex beat Reviso on DREAM-1K F1, suggesting open questions about which videos or harnesses still favour generic agentic workflows over a learned observation policy.

Target Audience

Researchers and engineers working on multimodal LLMs, video understanding, and agentic perception — particularly those building captioning or video-description systems and those applying reinforcement learning to tool-use policies. It is also relevant to practitioners who need to convert expensive proprietary-model outputs into supervision for smaller deployable models, and to readers interested in evidence-grounded generation and hallucination reduction. Readers without background in MLLM training or policy-gradient methods will find the framework description accessible but the reward formulation and experimental tables demanding.

Authors’ abstract

Multimodal large language models (MLLMs) can generate detailed video captions, yet they may overemphasize minor details while overlooking essential context and key events. We argue that not all visual evidence deserves equal attention, as the importance of visual information depends on the content of each video. Motivated by this observation, we propose Reviso, an agentic video captioning framework that learns what to observe and how to gather relevant visual evidence. From a low-frame-rate global preview, Reviso constructs a hierarchical Context--Focus--Points (CFP) observation plan that identifies the video context, prioritizes key aspects, and defines concrete observation targets and corresponding tools. Guided by this plan, Reviso performs goal-directed active perception by selectively revisiting temporal segments and combining tools for fine-grained inspection. It interprets the acquired evidence, updates its plan as needed, and integrates global and local information to generate the final caption. To learn this observation strategy, we develop a verification-guided pipeline that constructs high-quality training trajectories through factual verification and teacher-guided refinement. We further train Reviso using Group Relative Policy Optimization (GRPO), incorporating a key-fact reward for caption quality, a Plan Alignment Reward (PAR) for identifying important observation targets, and an Unproductive Tool-Call Penalty (UTCP) that adjusts observation-cost penalties based on caption quality. Experiments across four video captioning benchmarks show that Reviso outperforms the evaluated open-source baselines, while its generated captions better support downstream video question answering. These findings highlight the effectiveness and potential of priority-guided active observation for video captioning.