TimeLens: Rethinking Video Temporal Grounding with Multimodal LLMs

AI summary

Overview

Research area: Video understanding by multimodal large language models (MLLMs), specifically video temporal grounding (VTG) — the task of locating the time segment in a video described by a text query.

Technical level: Intermediate. The paper is written as a systematic empirical study, so the high-level argument is accessible, but full engagement requires familiarity with supervised fine-tuning, reinforcement learning with verifiable rewards (RLVR), GRPO, and timestamp encoding schemes used in video MLLMs.

Scope in one sentence: The paper does not propose a novel architecture; it diagnoses data-quality flaws in existing VTG benchmarks and training sets, releases corrected replacements (TimeLens-Bench and TimeLens-100K), and derives practical design rules that produce the TimeLens family of models.

What This Paper Is About

MLLMs are good at describing what happens in a video but poor at saying when it happens. The authors argue that progress on this problem has been held back less by model architecture than by unreliable data: existing VTG benchmarks contain ambiguous queries, duplicated queries, events that never occur, and imprecise timestamps, and the training corpus is even noisier. The goal is therefore to build a clean evaluation suite and a clean training set, then use them as a controlled testbed to determine which timestamp encoding and training-recipe choices actually work.

Key Contributions

1. A quality audit and rebuild of VTG evaluation. The authors define explicit criteria for queries (clarity, event existence, uniqueness, no positional leakage) and for segments (boundary precision, exhaustiveness), then manually re-annotate Charades-STA, ActivityNet Captions, and QVHighlights to produce TimeLens-Bench — a suite of 4,279 videos, average duration 107.8 seconds, and 9,404 annotations, with 6,463 queries rewritten and 2,941 time segments refined.

2. A corrected training corpus. Using an automated re-annotation pipeline (chosen because the training data showed an even higher error rate than the benchmarks), they produce TimeLens-100K, a large-scale, high-quality VTG training set.

3. A systematic study of algorithmic design choices. The paper compares position-embedding, visual-overlay, and textual timestamp encodings; compares SFT, thinking-based RLVR, thinking-free RLVR, and SFT-then-RLVR; and examines RLVR training length and data difficulty.

4. The TimeLens model family. Models built from Qwen2.5-VL-7B and Qwen3-VL-8B that set state-of-the-art VTG performance among open-source models and surpass proprietary models including GPT-5 and Gemini-2.5-Flash on TimeLens-Bench.

Main Findings

• Legacy benchmarks are broadly unreliable. In Charades-STA, 20.6% of samples violate query uniqueness and 34.9% have annotation accuracy problems. All three studied benchmarks show consistently high overall error rates, with the composition of error types differing across datasets.

• Correcting the benchmarks reverses model rankings. On the original benchmarks, proprietary models such as Gemini-2.5-Pro scored poorly while open-source models scored higher; on TimeLens-Bench the trend reverses, with proprietary models performing better and open-source models degrading substantially. The authors interpret this as evidence that the original benchmarks inflated open-source results and underestimated proprietary ones.

• Better training annotation yields stronger grounding. Training on TimeLens-100K instead of the original noisy data raises mIoU on Charades-TimeLens from 35.6 to 48.3, on ActivityNet-TimeLens from 31.3 to 43.1, and on QVHighlights-TimeLens from 44.6 to 56.7. Because the automated training re-annotation was developed independently of the manual benchmark refinement, the evaluation is unbiased with respect to the training pipeline.

• Interleaved textual timestamps win. Encoding each frame's timestamp as text tokens inserted before that frame's visual tokens, using raw timestamps (e.g., "10.2s") rather than frame indices, outperforms position-embedding-based methods and visual timestamp overlays while remaining simple.

• Thinking-free RLVR is the best training paradigm. It beat SFT with 32K data, SFT with 100K data, thinking-based RLVR, and SFT followed by thinking-free RLVR, while requiring the least training time (1.0x, where 1.0x is roughly 4h10m on 8 H20 GPUs). Adding an SFT phase before RLVR produced no significant gain.

• Early stopping prevents degradation. Tracking reward and evaluation metrics during RLVR, the authors find performance peaks when the temporal IoU reward and the within-group reward standard deviation plateau; training past that point degrades performance, so a full epoch over all data is suboptimal even with high-quality data.

• Difficult training samples matter. Performance improves as the average difficulty of RLVR training samples increases, plateauing when difficulty becomes sufficiently high (over 0.75). Difficulty is estimated by having the model to be trained perform offline inference on the training data and computing IoU.

• Thinking during RLVR is shallow. The authors observe that the model's thinking length gradually decreases during thinking-based RLVR and that the generated thinking is simple and primarily perception-related, without complex reasoning — consistent with VTG being a perception-driven task.

• Final results. TimeLens-7B improves substantially over the Qwen2.5-VL-7B baseline and closes much of the gap with Qwen3-VL-8B, surpassing models including Time-R1-7B, MiMo-VL-7B, GPT-4o, and GPT-5. TimeLens-8B reaches 76.6 / 63.0 / 35.2 with 55.2 mIoU on Charades-TimeLens, 68.9 / 58.4 / 40.6 with 53.2 mIoU on ActivityNet-TimeLens, and 80.2 / 71.6 / 55.5 with 65.5 mIoU on QVHighlights-TimeLens, exceeding Gemini-2.5-Flash.

Methodology in Plain English

The work proceeds in two tracks that reinforce each other.

Data track. The authors first write down what a good VTG annotation must satisfy — the query must be clear, refer to an event that actually occurs, be unique within its video, and not leak its own temporal position; the time segment must be tight and must cover every occurrence of the queried event. Because these abstract rules are hard to apply directly, they translate them into concrete, recognizable error types (multiple occurrences, no occurrence, duplicate queries, unclear query, inaccurate annotation) and run a "diagnose-then-refine" workflow in which the same annotator both finds the error and fixes it. Every batch is then cross-checked by a different annotator, and a batch exceeding an error threshold is rejected and re-annotated. The vendor was selected by trial annotation with over a dozen vendors.

Algorithm track. With the clean benchmark and clean training set in hand, they run controlled ablations on top of the Qwen2.5-VL-7B baseline, changing one design choice at a time against the best final configuration. They compare three families of timestamp encoding, four training paradigms under equal data and measured wall-clock budgets, and two RLVR recipes (when to stop, which samples to train on). GRPO is used as the RL optimizer, and approximately 12K samples selected from TimeLens-100K by difficulty-aware Gaussian sampling are used for the RLVR recipe experiments. The final recipe is then applied to two different base models to produce TimeLens-7B and TimeLens-8B.

Why This Matters

Impact on research. The paper argues that a meaningful part of the reported progress in VTG has been an artifact of noisy ground truth. By showing that cleaning the labels re-ranks models — reversing the apparent open-source versus proprietary ordering — it challenges the validity of prior leaderboards and provides a corrected suite and protocol (reporting metrics separately on the three constituent benchmarks) that future work can adopt. It also provides a rare controlled comparison of SFT versus RLVR under equal data and time budgets.

Real-world applications:

• Video search and retrieval, where a user describes a moment and expects the correct clip returned rather than the whole video.
• Video editing and content production, where a described event must be located automatically for cutting or captioning.
• Surveillance and security review, where an operator needs to jump to the moment a described activity occurred in hours of footage.
• Assistive and accessibility tools, where a spoken or written description must be mapped to the right moment of a long recording.

Industry relevance. The paper's practical claim is that strong temporal grounding comes from data hygiene plus a comparatively simple training recipe (thinking-free RLVR with early stopping and difficulty-based sampling) rather than from elaborate architecture changes. That is directly relevant to teams that need to post-train existing MLLMs on limited GPU budgets, and the reported 1.0x training time — roughly 4h10m on 8 H20 GPUs — makes the recipe concrete. The authors state that all code, data, and models will be released.

Future Directions

• Extending the audit beyond three benchmarks. TimeLens-Bench covers only refined Charades-STA, ActivityNet Captions, and QVHighlights; whether the same error rates and the same ranking reversals appear in other VTG datasets is an open question.
• Validating automated re-annotation at larger scale. The training corpus was fixed automatically while the benchmark was fixed manually; the paper reports the resulting gain but leaves open how far automated re-annotation can substitute for human annotation in general.
• Understanding why position-embedding methods underperform. The authors note these methods require fundamental modifications to RoPE and are impractical without large-scale retraining, but do not establish whether a properly retrained variant could close the gap.
• Explaining and exploiting the role of difficulty. The finding that difficult samples help and that gains plateau above roughly 0.75 difficulty suggests further work on how difficulty should be estimated and scheduled over the course of training, and on why explicit thinking collapses during RLVR for this perception-oriented task.

Target Audience

Researchers and engineers working on video-language models, video temporal grounding, or multimodal post-training workflows. It is most directly useful to practitioners who are fine-tuning or RL-tuning MLLMs for grounded video tasks and need to know which design choices are worth the compute, and to benchmark builders who want a template for diagnosing annotation quality. Readers looking for a new model architecture or a new RL algorithm will not find one here — the paper explicitly frames itself as an incremental but essential baseline.

Authors’ abstract

This paper does not introduce a novel method but instead establishes a straightforward, incremental, yet essential baseline for video temporal grounding (VTG), a core capability in video understanding. While multimodal large language models (MLLMs) excel at various video understanding tasks, the recipes for optimizing them for VTG remain under-explored. In this paper, we present TimeLens, a systematic investigation into building MLLMs with strong VTG ability, along two primary dimensions: data quality and algorithmic design. We first expose critical quality issues in existing VTG benchmarks and introduce TimeLens-Bench, comprising meticulously re-annotated versions of three popular benchmarks with strict quality criteria. Our analysis reveals dramatic model re-rankings compared to legacy benchmarks, confirming the unreliability of prior evaluation standards. We also address noisy training data through an automated re-annotation pipeline, yielding TimeLens-100K, a large-scale, high-quality training dataset. Building on our data foundation, we conduct in-depth explorations of algorithmic design principles, yielding a series of meaningful insights and effective yet efficient practices. These include interleaved textual encoding for time representation, a thinking-free reinforcement learning with verifiable rewards (RLVR) approach as the training paradigm, and carefully designed recipes for RLVR training. These efforts culminate in TimeLens models, a family of MLLMs with state-of-the-art VTG performance among open-source models and even surpass proprietary models such as GPT-5 and Gemini-2.5-Flash. All codes, data, and models will be released to facilitate future research.