LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Calling

AI summary

Overview

• Research area: Multimodal large language models (LMMs), long-video understanding and reasoning, tool-augmented agentic reasoning, and reinforcement learning for vision-language models.
• Technical level: Intermediate. The paper assumes familiarity with LMMs, supervised fine-tuning, GRPO-style reinforcement learning, and temporal grounding metrics such as IoU.
• Scope: The paper introduces LongVT, an end-to-end agentic framework that lets an LMM reason over hours-long videos by natively calling a video-cropping tool within an interleaved multimodal chain of thought, together with the VideoSIAH training and evaluation data suite and a three-stage training pipeline.

What This Paper Is About

Long-form videos (the paper defines these as longer than 15 minutes) contain the evidence needed to answer a question in only a few sparse, temporally dispersed moments, and existing LMMs tend to hallucinate when asked to reason over them using text-only chain-of-thought and uniform frame sampling. LongVT lets a model "think with long videos" instead of about them: it first skims the video globally, then repeatedly calls a native crop_video(start_time, end_time) tool to zoom into a proposed time window, resample finer-grained frames, and either refine its hypothesis or answer. To make this trainable and measurable, the authors build and release the VideoSIAH data suite and a 652-question human-validated benchmark (VideoSIAH-Eval) targeting a "video segment-in-a-haystack" setting.

Key Contributions

1. An end-to-end agentic paradigm for long-video reasoning. LongVT natively interleaves multimodal tool-augmented chain-of-thought with on-demand clip inspection over hours-long videos, exploiting the LMM's own latent temporal grounding ability as the cropping tool, without an auxiliary expert model or external retriever.
2. The VideoSIAH data suite and VideoSIAH-Eval benchmark. A semi-automatic, human-in-the-loop pipeline produces 247.9K SFT samples for tool-integrated cold start, 1.6K samples for agentic RL, and 15.4K samples for agentic reinforcement fine-tuning, plus a 652-QA-pair open-ended evaluation benchmark (244 videos) in which each question's evidence lies in a narrow window relative to the full video.
3. A three-stage training strategy. Cold-start SFT for tool-calling competence, GRPO-based agentic RL with a joint answer-temporal grounding reward, and agentic RFT that distills filtered, high-quality rollout traces back into supervised training.
4. Systematic ablation validation. The paper reports ablations on the data recipe, the training stages, and the temporal grounding reward design, and states it is the first to systematically show that the joint answer-temporal grounding reward and RFT each yield consistent gains over SFT or RL alone for long-video reasoning.

Main Findings

• State-of-the-art among open-source video-centric LMMs. LongVT variants are compared against Qwen2.5-VL-7B, Video-R1-7B, VideoRFT-7B, Video-Thinker-7B, GPT-4o and Gemini 1.5 Pro on VideoMME, VideoMMMU, LVBench and VideoSIAH-Eval, under both sparse (64 frames) and dense (512 or 768 frames, better reported) sampling.
• Dense-sampling results. LongVT-7B-RFT scores 67.0 on VideoMME, 35.7 / 43.7 / 56.7 on VideoMMMU adaptation / comprehension / perception, 41.3 on LVBench, 42.0 on VideoSIAH-Eval, for an average of 47.7. LongVT-7B-RL scores 66.1 / 37.7 / 42.3 / 56.3 / 41.4 / 35.9, averaging 46.6.
• The hardest benchmark shows the largest gap. On VideoSIAH-Eval, LongVT-7B-RFT reaches 42.0, outperforming the second-best model by 6 points; GPT-4o scores 17.4 and Gemini 1.5 Pro's result on that column is not reported ("-").
• Proprietary gap narrowed. The best LongVT checkpoint lies within roughly four points of GPT-4o on average, based on GPT-4o's reported average of 51.5.
• Sparse-sampling results. LongVT-7B-RL at 64 frames slightly surpasses the best existing open-source baseline, with 66.1 on VideoMME, 32.7 / 44.7 / 50.0 on VideoMMMU, 37.8 on LVBench and 31.0 on VideoSIAH-Eval (average 43.7).
• Self-curated data matters. Removing self-curated iMCoTT traces from SFT drops the average from 44.1 to 24.8 (including VideoMME falling to 8.4 and VideoSIAH-Eval to 4.1). Removing self-curated QAs from RL drops the average from 46.6 to 40.4.
• All three stages are needed. RL only scores lowest across all four benchmarks (average 41.9), SFT only reaches 44.1, SFT+RL 46.6, and SFT+RL+RFT 47.7.
• IoU reward beats Recall reward for grounding. On Charades-STA, IoU-rewarded RL achieves 41.0 IoU@0.3, 25.8 IoU@0.5, 11.7 IoU@0.7 and 27.2 mIoU, versus 32.0 / 20.4 / 9.6 / 21.6 for Recall reward and 31.5 / 19.9 / 9.1 / 21.2 without a decoupled temporal grounding reward.
• Recall reward invites reward hacking. The paper reports the policy can inflate the predicted span to envelop the ground-truth interval, monotonically raising the Recall-based score while ignoring boundary quality.
• Tool reward is not needed. The Qwen2.5-VL-7B baseline collapses to near-zero tool calls in both configurations (with and without tool reward); after cold-start SFT, tool-call frequency rises during training and accuracy improves in tandem, so the authors discard the tool reward and rely on accuracy, format and decoupled IoU rewards.
• Cold start is indispensable. Preliminary RL with Qwen2.5-VL-7B showed the model failing to improve and ultimately collapsing, which the authors attribute to weak temporal localization and weak integration of tool outputs.
• No latency penalty. Despite multi-turn tool interactions, LongVT incurs no additional inference latency and can be faster than single-turn baselines by avoiding hallucination-driven verbose generation (details deferred to the supplementary material).
• Benchmark characteristics. VideoSIAH-Eval has an average video duration of approximately 1,688 seconds, with 71.84% of videos in the 15–30 minute range and 28.16% longer than 30 minutes; an earlier release contained 1,280 entries due to unintentional duplication during data export.

Methodology in Plain English

The authors start from a human analogy: when a person watches a long match to answer a specific question, they skim globally for strong visual cues, then rewind and inspect the decisive moment closely.

• Tool use. The model is taught to emit a crop_video(start_time, end_time) call, which crops the video and resamples finer-grained frames inside the proposed window. The model then re-reasons over the new evidence and either refines its window or answers. The ability to propose the window comes from the model's own temporal grounding, not from a separate detector.
• Data construction. Long videos are split using a deterministic, pixel-level scene detection algorithm, with consecutive segments shorter than 10 seconds merged. Qwen2.5-VL-72B writes detailed segment captions, which seed QA generation covering temporal events, spatial layout, motion, object attributes and scene transitions. Two filtering stages follow: text-based filtering for ill-posed or leaky questions, then GLM-4.5V-based multimodal verification of answer consistency against the video. Annotators review representative failures to refine prompting rules. Multi-round tool traces are generated with a probability that increases with video length, so longer videos get proportionally more tool-calling rounds.
• Training stage 1 — cold-start SFT. On 247.9K samples (228,835 non-tool plus 19,161 tool-augmented, of which 12,766 are Gemini-distilled iMCoTT traces for open-ended QA and 6,395 Qwen-distilled traces for temporal grounding), the base Qwen2.5-VL-7B learns to propose windows, call the tool and reason over resampled frames.
• Training stage 2 — agentic RL. GRPO is used with a joint answer-temporal grounding reward: an LLM-as-a-Judge accuracy score (1 for fully consistent, 0.5 for partially consistent, 0 for inconsistent), a binary format-compliance score, and the IoU between predicted and ground-truth spans. RL questions are length-balanced across short/medium/long duration bands, then filtered by difficulty — questions where all K rollouts succeed or all K fail are discarded, keeping only mixed outcomes.
• Training stage 3 — agentic RFT. Rollouts from early RL runs are kept only if the final answer is correct and the predicted temporal span reaches IoU of at least 0.3 with the ground-truth window. These 15,353 filtered traces are converted into supervised examples to stabilize the learned agentic behavior.

Why This Matters

• Impact on research. The work reframes long-video understanding as active, evidence-seeking reasoning rather than passive frame consumption, and provides an open data suite, benchmark, code and model weights. It also reports a contamination study motivating open-ended evaluation: Qwen2.5-VL-7B-Instruct still scores 40.1 on VideoMME and 12.7 on VideoSIAH-Eval with no visual input, and Qwen3-VL-8B-Instruct drops to 0.00 on VideoSIAH-Eval with no visual input, whereas the authors state the rearranged-choices perturbation is not applicable to their fully open-ended benchmark.
• Real-world applications (as grounded in the paper's cited use cases):
  • Soccer event spotting, such as locating the moment and manner of a specific goal.
  • Film and long-form content understanding.
  • Video question answering over hours-long footage.
  • Temporal grounding and dense captioning pipelines that need to point at specific moments.
• Industry relevance. Fine-grained temporal localization and grounded QA are the building blocks for search, indexing, compliance review and analytics over long video archives. The paper's claim of no additional inference latency for multi-turn tool use matters for deployment economics, and the finding that results stay within roughly four points of GPT-4o on average suggests open-weight 7B models can approach proprietary performance on long-video tasks. The project was fully supported by MiroMind, which provided compute, storage and engineering infrastructure.

Future Directions

• Architectural limitations and multi-agent extensions. The supplementary material outlines a

Authors’ abstract

Large multimodal models (LMMs) have shown great potential for video reasoning with textual Chain-of-Thought. However, they remain vulnerable to hallucinations, especially when processing long-form videos where evidence is sparse and temporally dispersed. Inspired by how humans comprehend long videos - by first skimming globally and then examining relevant clips for details - we introduce LongVT, an end-to-end agentic framework that enables "Thinking with Long Videos" via interleaved Multimodal Chain-of-Tool-Thought. Specifically, we exploit LMMs' inherent temporal grounding ability as a native video cropping tool to zoom in on a specific video clip and resample finer-grained video frames. This global-to-local reasoning loop continues until answers are grounded in retrieved visual evidence. Given the scarcity of fine-grained question-answering (QA) data for the long video reasoning task, we curate and will release a data suite named VideoSIAH to facilitate both training and evaluation. Specifically, our training dataset consists of 247.9K samples for tool-integrated cold-start supervised fine-tuning, 1.6K samples for agentic reinforcement learning, and 15.4K samples for agentic reinforcement fine-tuning, respectively. Our evaluation benchmark consists of 1,280 QA pairs that are carefully curated through a semi-automatic data pipeline with human-in-the-loop validation. With a meticulously designed three-stage training strategy and extensive empirical validation, LongVT consistently outperforms existing strong baselines across four challenging long-video understanding and reasoning benchmarks. Our codes, data, and model checkpoints are publicly available at https://github.com/EvolvingLMMs-Lab/LongVT .