Enrich and Detect: Video Temporal Grounding with Multimodal LLMs

AI summary

Overview

Research area: Computer vision / video-language understanding, specifically video temporal grounding with multimodal large language models (LLMs).

Technical level: Advanced. The paper assumes familiarity with multimodal LLMs, transformer detectors, regression losses (L1, gIoU), and multiple-instance learning.

Scope: The paper proposes ED-VTG, a two-stage "enrich-then-detect" framework that uses a multimodal LLM to rewrite vague language queries into more detailed ones and a lightweight interval decoder to localize them in untrimmed video, reporting results across single-query grounding, video paragraph grounding, question grounding, and article grounding benchmarks.

What This Paper Is About

Video temporal grounding asks a model to find the time interval in a video that matches a text query. The authors observe that the queries in existing grounding datasets are often short, coarse, or incomplete, which limits how precisely a model can localize them. Their goal is to let a multimodal LLM first enrich a query with details drawn from the video, and then detect the corresponding temporal boundaries from the enriched query, rather than grounding the original query directly.

Key Contributions

1. A cascaded enrich-and-detect approach to temporal grounding, where a multimodal LLM first transforms the input query into an enriched sentence using video context, and the enriched query is then localized. The chain (Video, Query) → enriched query, then (Video, enriched query) → interval, is described as the key idea of the method.
2. A lightweight interval decoder that lets multimodal LLMs localize text queries precisely. The decoder regresses an interval parameterized by center and width (chosen for decoupling position from scale), which allows training with detection objectives such as L1 and generalized IoU (gIoU) with minimal added complexity.
3. A multiple-instance learning (MIL) training paradigm that dynamically selects, per training sample, whether the original query or the enriched query leads to a better interval prediction, mitigating noise and hallucinations in the pseudo-labeled enriched queries.
4. State-of-the-art results across multiple temporal grounding benchmarks for both single-query and paragraph grounding, reported as the first LLM-based model that surpasses or performs comparably to specialist models.

Main Findings

• Zero-shot single-query grounding improves over all prior methods. On Charades-STA, ED-VTG reaches R@0.3 = 59.5, R@0.5 = 39.3, R@0.7 = 19.8, mIoU = 40.2; on ActivityNet-Captions, 52.1 / 33.1 / 16.0, mIoU 35.2; on TACoS, 14.5 / 6.0 / 2.3, mIoU 12.7. The reported deltas versus HawkEye are +8.9, +7.9, +5.3, +6.5 on Charades and +3.0, +3.8, +5.3, +2.5 on ActivityNet; versus ChatVTG on TACoS they are +6.4, +2.3, +1.0, +7.2.
• Enrichment helps most where queries are vague. The paper reports that ED-VTG surpasses Momenter by 11.4 and HawkEye by 6.2 absolute mIoU points on Charades despite those models being pre-trained with 100x and 6x more segment-level data, and gains 2.5 and 7.2 mIoU points over the nearest LLM-based models on ActivityNet and TACoS in the zero-shot setting.
• Fine-tuned single-query results are also strong. On Charades-STA the model reaches 78.2 / 62.1 / 35.0, mIoU 52.6; on ActivityNet-Captions 67.6 / 45.1 / 22.7, mIoU 44.9; on TACoS 46.0 / 31.5 / 15.8, mIoU 32.4. Gains over HawkEye are 5.7 and 11.7 R@0.3 points on Charades and ActivityNet, and 14.0 and 14.4 mIoU points over TimeChat and VTimeLLM on TACoS.
• Video paragraph grounding is addressed by an LLM-based model for the first time. On Charades-CD-OOD it reaches 70.7 / 47.3, mIoU 45.0 (a 9.9 and 6.7 mIoU gain over VTimeLLM and TimeChat); on ActivityNet-Captions 74.1 / 58.0, mIoU 53.7; on TACoS 46.2 / 27.8, mIoU 30.7; on YouCook2 48.1 / 28.0, mIoU 31.5.
• Question grounding results. On the NExT-GQA test split, ED-VTG reports mIoP 34.7, IoP@0.3 45.1, IoP@0.5 33.5, mIoU 26.6, IoU@0.3 39.5, IoU@0.5 19.8, with deltas over SeViLA of +5.2, +10.4, +10.6, +4.9, +10.3, +6.0, and over HawkEye of +0.9 mIoU, +2.5 IoU@0.3, +0.3 IoU@0.5.
• Article grounding with non-groundable queries. On HT-Step, ED-VTG is reported as the first LLM-based model to report results for grounding in the presence of negative, non-groundable queries, with seen-split mAP@0.3/0.5/0.7 of 48.9 / 31.5 / 18.0 (average mAP@[0.3-0.7] 32.5) and unseen-split 33.0 / 21.2 / 11.1 (average 21.6), improving over TimeChat by 3.6 / 2.5 / 3.6 / 3.5 on seen and 2.3 / 3.4 / 3.6 / 2.9 on unseen.
• Enrichment is the main driver of gains. In the fine-tuning-without-pretraining setting on Charades-STA, grounding alone gives 51.4 / 31.5 / mIoU 33.2, enrich-and-detect gives 60.1 / 37.0 / 38.4, and adding MIL gives 62.8 / 38.4 / 40.3. On ActivityNet-Captions the same progression is 50.3 / 30.1 / 34.0 → 56.3 / 35.5 / 37.8 → 57.5 / 36.2 / 38.6.
• Offline enrichment is not enough. Replacing the learned two-stage pipeline with a pre-processing step that enriches training queries offline yields 51.7 / 31.5 / 33.4 on Charades-STA and 49.8 / 29.9 / 33.7 on ActivityNet-Captions, far below the 60.1 / 37.0 / 38.4 and 56.3 / 35.5 / 37.8 of enrich-and-detect.
• The interval decoder and detection losses matter. Training only with the language-model loss (no decoder) gives 54.2 / 33.2 / 34.1 on Charades-STA and 51.0 / 31.6 / 35.5 on ActivityNet-Captions, while combining LM loss with L1 and gIoU gives 60.1 / 37.0 / 38.4 and 56.3 / 35.5 / 37.8.

Methodology in Plain English

The system has three parts: a vision encoder that turns video frames into visual tokens, a multimodal LLM that reads the video tokens and the text query together, and a small interval decoder.

Stage one (enrich). The LLM is prompted to rewrite the user's query into a more detailed sentence using what it sees in the video. For example, a vague query like "Man starts surfing" is refined into something like "The man with a yellow surfboard slowly runs to start surfing." After producing the enriched text, the LLM emits a special token, <INT>, signalling that it is ready to localize.

Stage two (detect). The hidden state of that <INT> token is projected and fed, together with the visual tokens, into the interval decoder — two transformer layers followed by a multi-layer perceptron. The decoder outputs a predicted interval expressed as a center and a width. Because the <INT> token is generated after the enriched query, the quality of the enrichment directly affects the interval prediction; this cascaded dependency is the central design choice.

Training. There are no human annotations for "good" enriched queries, so the authors generate pseudo-labels by prompting an off-the-shelf captioning model (LLaVA OneVision 72B) with the original query and the ground-truth video segment. Because these pseudo-labels can contain hallucinations, training uses multiple-instance learning: for each sample, the model runs two forward passes — one with the original query and one with the enriched query — producing two interval predictions. The version with the lower grounding loss is the one backpropagated through, so the model learns when enrichment is useful and when the original query is already good enough. The loss combines a language modeling cross-entropy term and a grounding term made of L1 plus gIoU.

Setup. The video encoder and LLM are initialized from the Video-LLaMA-7B checkpoint (ViT-G/14 from EVA-CLIP plus image and video Q-Formers). The ViT is frozen, LoRA with rank 32 is applied to the LLM, and the Q-Formers, decoder, and linear layers are fully tuned. Pre-training uses 136K videos from 8 public datasets for 40 epochs with batch size 256, AdamW, peak learning rate 5e-5, and a cosine schedule with linear warmup over the first 20% of steps, taking 2 days on 16 V100 nodes (8 cards with 32G GPU memory each). For video paragraph grounding, four forward passes are used to perform MIL.

Tasks covered. Single-query temporal grounding (STG), video paragraph grounding (VPG), question grounding (QG, evaluation only), and article grounding (AG, where some queries may not be groundable). Metrics are mIoU and Recall@1 for IoU thresholds of 0.3, 0.5 and 0.7; for QG, intersection over prediction (IoP) plus IoU following the NExT-GQA protocol; for AG, mAP across IoU thresholds following the HT-Step protocol. Evaluation is reported in zero-shot and fine-tuned settings.

Why This Matters

Impact on research. The paper argues that temporal grounding and video captioning are dual tasks — the output of one is the input to the other — and that this synergy has been largely unexplored, with prior work specializing in one task or training multi-task without studying cross-task benefit. ED-VTG is presented as the first LLM-based model to surpass or match specialist grounding models, and the ablation showing that offline enrichment underperforms a learned enrichment stage is a direct challenge to the idea that captioning is merely data augmentation.

Real-world applications:

• Video editing, where an editor describes a moment in words and the tool locates the exact clip boundaries.
• Content retrieval, finding the relevant segment of a long video from a natural-language description.
• Explainable video question answering, where question grounding produces evidence intervals supporting an answer.
• Long-form video indexing and chaptering, since paragraph grounding and article grounding localize many queries — and identify non-groundable ones — within a single video.

Industry relevance. The method uses an open-weight 7B backbone (Video-LLaMA-7B) with LoRA rather than a much larger closed model, and adds only a small decoder, which the authors present as achieving specialist-level accuracy at modest added complexity. The article-grounding results on HT-Step, where the model must also detect queries that cannot be grounded, point toward practical deployment on procedurally structured content such as cooking and instructional video.

Future Directions

• Reducing reliance on the external captioner. Enriched-query labels come from a separate 72B captioning model that has access to ground-truth intervals during training. Improving the quality of these pseudo-labels, or removing the need for a ground-truth-conditioned captioner, is a natural next step given the paper's own concern about hallucinations.
• Scaling the enrichment/detection paradigm to more queries and longer videos. The paper notes that datasets like TACoS, with short under-specified queries, long videos, and fine-grained intervals, challenge LLMs with a fixed number of input frames.
• Extending MIL beyond the binary original-versus-enriched choice. The framework currently chooses between two query versions for STG and runs four forward passes for VPG; whether broader candidate sets or learned selection policies help is left open.
• Broadening evaluation to more tasks and domains. The reported results cover STG, VPG, QG, and AG; whether the same enrich-and-detect principle transfers to other video-language tasks that share the grounding/captioning duality is an open question.

Target Audience

Researchers and graduate students in computer vision and video-language understanding who work on temporal grounding, video question answering, or multimodal LLM adaptation. It is also relevant to practitioners building video search, editing, or indexing systems who want to know how far an open 7B multimodal LLM plus a small detection head can go compared with task-specific architectures. Readers without background in multimodal LLMs, detection losses, or multiple-instance learning will find the method sections demanding; the high-level enrich-then-detect idea, however, is accessible to anyone familiar with video grounding.

Authors’ abstract

We introduce ED-VTG, a method for fine-grained video temporal grounding utilizing multi-modal large language models. Our approach harnesses the capabilities of multimodal LLMs to jointly process text and video, in order to effectively localize natural language queries in videos through a two-stage process. Rather than being directly grounded, language queries are initially transformed into enriched sentences that incorporate missing details and cues to aid in grounding. In the second stage, these enriched queries are grounded, using a lightweight decoder, which specializes at predicting accurate boundaries conditioned on contextualized representations of the enriched queries. To mitigate noise and reduce the impact of hallucinations, our model is trained with a multiple-instance-learning objective that dynamically selects the optimal version of the query for each training sample. We demonstrate state-of-the-art results across various benchmarks in temporal video grounding and paragraph grounding settings. Experiments reveal that our method significantly outperforms all previously proposed LLM-based temporal grounding approaches and is either superior or comparable to specialized models, while maintaining a clear advantage against them in zero-shot evaluation scenarios.