VideoTemp-o3: Harmonizing Temporal Grounding and Video Understanding in Agentic Thinking-with-Videos

AI summary

Overview

• Research area: Computer vision and multimodal large language models (MLLMs), specifically long-video understanding, temporal grounding, and agentic "thinking-with-videos" tool use.
• Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning (SFT), reinforcement learning (GRPO), video question answering (VideoQA), and temporal grounding metrics such as mIoU and R@0.7.
• Scope: This paper proposes VideoTemp-o3, a single unified model that decides on demand when to localize and crop video segments, can refine imprecise localizations over multiple turns, and answers grounded in the cropped evidence, supported by a new data curation pipeline and a new duration-stratified benchmark (VideoTemp-Bench).

What This Paper Is About

Long videos are usually processed by sampling frames uniformly under a fixed frame budget, which often misses the evidence a question actually needs and increases hallucination. Recent "thinking-with-videos" systems instead follow a localize–clip–answer loop, but the paper argues existing versions are inefficient (multiple specialized models), imprecise at grounding, and rigid (one blind crop, then answer). VideoTemp-o3 aims to unify temporal grounding and VideoQA inside one model with on-demand cropping and iterative refinement.

Key Contributions

1. A unified agentic model. VideoTemp-o3 harmonizes temporal grounding and VideoQA in one architecture, supports on-demand video cropping, and enables multi-turn grounding refinement.
2. New training strategies. A cold-start SFT stage with a unified masking mechanism, plus tailored RL rewards (including a penalty-aware IoU reward) designed to strengthen internal grounding and mitigate reward hacking.
3. A data curation pipeline. A pipeline that produces large-scale long-video grounded QA data with single-turn and multi-turn (tool-call) trajectories, where cropped segments are tightly aligned with answers.
4. A new benchmark. VideoTemp-Bench, built to evaluate grounded QA across video durations of 0~3 min, 3~10 min, 10~20 min, and >20 min, with 300 samples per category and 1,200 samples in total.

Main Findings

• Long video understanding gains. VideoTemp-o3-7B-RL reports 54.2 on MLVU (M-Avg), 53.2 on VideoMMMU (Avg), 64.5 on VideoMME (w/o subtitle, Overall), and 43.0 on LVBench. The paper states this corresponds to 2.4% and 1.7% improvements on VideoMME and LVBench, and that RL adds roughly 4~5% over SFT across all benchmarks.
• Strong internal grounding. On temporal grounding, VideoTemp-o3-7B-RL reaches 33.0 R@0.7 and 57.8 mIoU on Charades-STA, and 26.7 R@0.7 and 45.3 mIoU on ActivityNet-MR, described as comparable to expert grounding models such as TimeMaker.
• Better grounded QA. On NextGQA the model reports 33.4 mIoU and 76.4 accuracy; on ReXTime it reports 29.5 mIoU and 74.4 accuracy. Higher mIoU tracks with higher answer accuracy.
• Grounding data helps QA too. Removing grounding data from SFT (variant b) drops ReXTime from 29.5 to 13.0 mIoU and from 74.4 to 73.3 accuracy, a stated gap of 16.5% and 1.1%.
• Unified masking is important. Supervising all turns instead of only the final two (variant c) degrades VideoMMMU to 47.9, VideoMME to 61.5, LVBench to 41.2, and ReXTime to 18.8 mIoU / 70.6 accuracy.
• Naive IoU rewards invite reward hacking. Replacing the penalty-aware IoU reward with a plain IoU reward raises the tool-call (clipping) ratio sharply while degrading grounding quality; removing IoU rewards entirely (variant d) causes a modest decline, including 1.6% on VideoMMMU.
• Duration is the bottleneck. On VideoTemp-Bench, grounding and QA degrade as videos get longer: Qwen2.5-VL drops from 70.3% to 49.1% accuracy between the 10~20 min and >20 min categories. Gemini-2.5-Pro's mIoU falls from 46.1 in the 3~10 min bucket to 14.8 in the >20 min bucket.
• Task difficulty varies widely. Across video tasks, complex reasoning (temporal reasoning) and fine-grained perception (counting, OCR) are hardest, while information synopsis and recognition are easier; the paper reports an approximately 40% gap between counting and recognition tasks.
• Learned tool use beats prompt tricks. On VideoMME (w/o subtitle), prompt-only re-grounding scores 58.4 and inference-time self-correction 60.5, versus 64.5 for VideoTemp-o3; cropping once (63.0) and no tool call (62.1) also underperform.
• On-demand behavior is real. Tool-call rates and average number of clips per video rise with video length, showing the model crops more when evidence is sparse.

Methodology in Plain English

The model treats "thinking with videos" as a repeated loop: given a video and a question, it skims at a low sampling rate, writes reasoning, and either emits a temporal interval or a final answer. If it emits an interval, an external cropping module extracts that clip at a higher sampling rate and appends it to the context for the next turn. The loop ends when the model answers or hits a maximum number of turns.

Training happens in two stages. First, cold-start SFT teaches the localize–clip–answer pattern using curated single-turn and multi-turn trajectories. A key design is the unified masking mechanism: in multi-turn dialogues, only the final two turns of model output are supervised, because earlier intervals are typically imprecise and supervising them adds noise. Second, agentic RL with GRPO optimizes whole trajectories using three rewards: an accuracy reward (1 if the answer matches ground truth), a format reward (1 if the dialogue format matches), and a penalty-aware IoU reward that subtracts a penalty of 0.1 when IoU falls below a threshold of 0.1, to stop the model from guessing intervals to game the reward.

The data pipeline builds training material with teacher models. Single-turn QA reasoning chains come from Qwen3-VL-235B-A22B-Thinking, keeping only samples where the predicted answer matches ground truth; grounding annotations are re-localized by Gemini-2.5-Pro and kept only when the predicted interval reaches IoU > 0.5 with the original annotation. Multi-turn data uses a two-step verification: first, crop a candidate segment predicted by Gemini-2.5-Pro and require the model to answer correctly using only that segment; second, run a closed-loop consistency check with the full context. Failures are re-grounded once and re-verified; samples passing the extra round become multi-tool-call data. This procedure is applied only to videos longer than 3 minutes. RL annotations undergo human verification, with all valid evidence intervals marked when a question can be answered from multiple segments. The backbone is Qwen2.5-VL-7B, trained with ms-swift and vLLM.

Why This Matters

Impact on research. The paper argues that precise temporal grounding is a prerequisite for grounded video reasoning, and that unifying grounding and QA in one model plus explicit trajectory data is more effective than prompt-level interventions or rigid single-crop pipelines. It also contributes a duration-stratified benchmark that exposes how sharply performance falls on hour-scale videos, which the authors claim current benchmarks (mostly under 3 minutes) do not reveal.

Real-world applications (drawn from the paper's impact statement):

• Video retrieval, where a query must be matched to the right moment in a long recording.
• Educational content analysis, such as locating a specific explanation inside a lecture.
• Accessibility tools that help users navigate and review long-form visual records.
• Assistance for reviewing long-form visual material more generally.

Industry relevance. The work is a collaboration between Shandong University, the Institute of Automation of the Chinese Academy of Sciences, Beihang University, Southern University of Science and Technology, and Kuaishou Technology, with funding acknowledged from Kuaishou (No. DJHL-20240801-004). The paper notes risks: more capable video understanding can be misused for intrusive surveillance, privacy-sensitive profiling, or automated decisions without human oversight, and warns the method should not be deployed directly in high-stakes settings, since it can still produce wrong localizations or answers, especially on very long videos, ambiguous questions, or events lasting only a few frames.

Future Directions

• Add broader tools. The authors suggest incorporating search engines and spatiotemporal grounding operations so the model can handle complex and up-to-date tasks.
• Fix sparse initial skimming. For very long videos and extremely short events (a few frames), the first low-rate skim can miss key evidence before refinement; the paper proposes hierarchical sampling and finer temporal modeling as possible remedies.
• Reduce sequential overhead. The localize–clip–answer process adds inference cost; the paper suggests parallel clip verification as a direction.
• Weaken the masking assumption. The unified masking strategy assumes later verified turns are more reliable than earlier exploratory ones; if future data curation can supervise every turn accurately, the heuristic may need to change, possibly toward weaker forms of grounding supervision.

Target Audience

Researchers and engineers working on multimodal LLMs, long-video understanding, video question answering, and temporal grounding, as well as practitioners building agentic video systems that must decide when to look closer. It is also relevant to readers interested in RL reward design for multimodal agents and in benchmark construction for duration-sensitive video evaluation.

Authors’ abstract

In long-video understanding, conventional uniform frame sampling often fails to capture key visual evidence, leading to degraded performance and increased hallucinations. To address this, recent agentic thinking-with-videos paradigms have emerged, adopting a localize-clip-answer pipeline in which the model actively identifies relevant video segments, performs dense sampling within those clips, and then produces answers. However, existing methods remain inefficient, suffer from weak localization, and adhere to rigid workflows. To solve these issues, we propose VideoTemp-o3, a unified agentic thinking-with-videos framework that jointly models video grounding and question answering. VideoTemp-o3 exhibits strong localization capability, supports on-demand clipping, and can refine inaccurate localizations. Specifically, in the supervised fine-tuning stage, we design a unified masking mechanism that encourages exploration while preventing noise. For reinforcement learning, we introduce dedicated rewards to mitigate reward hacking. Besides, from the data perspective, we develop an effective pipeline to construct high-quality long video grounded QA data, along with a corresponding benchmark for systematic evaluation across various video durations. Experimental results demonstrate that our method achieves remarkable performance on both long video understanding and grounding.