Empower Words: DualGround for Structured Phrase and Sentence-Level Temporal Grounding

AI summary

Overview

Research area: Computer vision, specifically video-language understanding — Video Temporal Grounding (VTG), covering Moment Retrieval (MR) and Highlight Detection (HD).

Technical level: Advanced. Familiarity with vision-language models (CLIP, InternVideo2), cross-attention, DETR-style decoding, and Slot Attention is needed to follow the method details.

Scope: The paper diagnoses an over-reliance on the sentence-level [EOS] token in existing VTG models and proposes DualGround, a dual-branch architecture that models sentence-level and phrase-level semantics separately, evaluated on QVHighlights and Charades-STA.

What This Paper Is About

Video Temporal Grounding asks a model to find the moment in a long, untrimmed video that matches a natural-language query, and to score which clips are highlights. The authors show that current models concentrate attention on the [EOS] token — a single embedding that summarizes the whole sentence — and largely ignore individual words, which hurts precise localization. DualGround addresses this by routing the [EOS] token down one path for global alignment and clustering word tokens into phrase units on a second path for localized, fine-grained grounding.

Key Contributions

1. An empirical diagnosis of [EOS] bias. Controlled experiments on the QVHighlights validation set using InternVideo2 features compare three input configurations — word tokens only, [EOS] only, and full sequences — and show models perform comparably or better with only the [EOS] token. Attention maps from FlashVTG show dominant focus on [EOS] even for clips unrelated to the query, with visually salient words such as "red jacket" underused.

2. A dual-path architecture (DualGround) that structurally separates global and local semantics: a sentence-level path that uses only the [EOS] token, and a phrase-level path that clusters word tokens into phrase units.

3. Token-role-aware cross-modal interaction, including Adaptive Cross Attention (ACA) with learnable dummy tokens as attention sinks on the sentence path, and a Recurrent Phrase Generator (RPG) plus Slot Attention refinement on the phrase path.

4. A joint training framework adding phrase-level supervision — a Distinct Query Attention (DQA) loss and an [EOS] reconstruction loss — alongside standard MR and HD objectives, with state-of-the-art results reported on QVHighlights and Charades-STA.

Main Findings

• [EOS] dominates existing models: Across the three input configurations tested on QVHighlights val with InternVideo2 features, performance with only the [EOS] token was comparable to or better than using the full token sequence. An appendix correlation analysis (Appendix B.1) is cited as showing consistently high alignment between [EOS] and word-token attentions in prior VTG models.

• QVHighlights test results (Table 1): With CLIP+SlowFast features DualGround reaches R1@0.5 of 68.20, R1@0.7 of 51.72, mAP of 49.02, mAP@0.5 of 69.23, mAP@0.75 of 47.71, VG-mAP of 41.15 and VG-Hit@1 of 66.30. With InternVideo2 features it reaches R1@0.5 of 71.87, R1@0.7 of 56.94, mAP of 52.73, mAP@0.5 of 72.41, mAP@0.75 of 54.38, VG-mAP of 44.02 and VG-Hit@1 of 70.80. The paper states the R1@0.7 improvement is 1.72% with CLIP+SlowFast and 2.98% with InternVideo2.

• QVHighlights validation results (Table 2), all with InternVideo2: DualGround scores 73.48 R1@0.5, 58.97 R1@0.7, 53.26 mAP, 72.99 mAP@0.5, 56.35 mAP@0.75, 44.12 VG-mAP and 71.62 VG-Hit@1. The strongest comparison listed, FlashVTG, scores 71.48, 56.06, 52.61, 72.37, 55.03, 44.08 and 71.48 respectively.

• Ablation on QVHighlights val (Table 3): The baseline setting (a), corresponding to FlashVTG with the full-token sequence, scores 56.13 R1@0.7, 52.24 mAP, 70.88 VG-Hit@1 and 44.04 VG-mAP. The full DualGround configuration (g), using RPG, Slot Attention, the DQA loss and the [EOS] loss, scores 58.97 R1@0.7, 53.26 mAP, 71.62 VG-Hit@1 and 44.12 VG-mAP. The paper reports that settings omitting phrase regulation (for example without the DQA or [EOS] loss) lead to slight degradations in HD metrics.

• Robustness to long queries (Table 4, QVHighlights val, IV2 backbone), MR mAP by query length: For queries over 20 words DualGround scores 48.92, versus 43.46 for FlashVTG, 35.33 for TR-DETR and 35.21 for CG-DETR. For the 0–10 bucket DualGround scores 54.12, for 10–15 it scores 54.33, and for 15–20 it scores 51.96.

• Charades-STA test results (Table 5): With CLIP+SlowFast features DualGround reaches R1@0.5 of 61.11 and R1@0.7 of 38.52; with InternVideo2 features it reaches 70.67 and 50.33. The strongest baseline listed, FlashVTG, scores 61.08 and 37.89 with CLIP+SlowFast, and 70.32 and 49.87 with InternVideo2.

• Qualitative result: On a QVHighlights validation example, baselines focus on a long "interviewing" scene and underuse the phrase "red jacket," while DualGround localizes the described moment and its phrase clusters align with localized visual cues.

• TVSum: TVSum is listed among the evaluation benchmarks and I3D+CLIP is described as the feature setup for it, but no TVSum result table appears in the content provided.

Methodology in Plain English

The model keeps the sentence-level and word-level signals on two separate tracks instead of mixing them into one flat token sequence.

On the sentence-level path, the model uses only the [EOS] token. Because a single token cannot support normal attention, the authors adopt Adaptive Cross Attention from CG-DETR: learnable "dummy" tokens are added to the key sequence and act as attractors that soak up attention from video clips that are not relevant to the query, so relevant clips attend more sharply to [EOS]. A stack of self-attention layers over the temporal dimension keeps the resulting video representation temporally coherent.

On the phrase-level path, the word tokens (excluding [EOS]) are grouped into a fixed number of phrases. A Recurrent Phrase Generator builds these phrases one at a time: each phrase is formed by a guide vector that attends over word tokens, conditioned on the [EOS] embedding and the previously generated phrase, with positional embeddings injected to encourage grouping of adjacent words. A Slot Attention module then refines the phrases, treating each as a latent slot that selectively aggregates word features. A learnable phrase token, P_[EOS], is appended so the phrase path can also form a global summary and produce importance weights for aggregation.

Each phrase is then compared with each video clip through a Hadamard product of projected representations, followed by temporal self-attention, producing a phrase-clip context tensor. These contexts are combined into a single phrase-guided video representation using attention weights derived from similarity to P_[EOS].

The two representations — sentence-guided and phrase-guided — are summed into a fused feature, which is decoded with the multi-scale framework from FlashVTG and R2-Tuning rather than standard DETR-style queries. A temporal feature pyramid built from stacked 1D convolutions allows moment predictions at multiple temporal resolutions, while highlight detection is performed at the base resolution using a saliency scoring head that combines global and local context via Hadamard interaction.

Training combines a Focal classification loss and an L1 boundary regression loss for moment retrieval, a highlight detection loss made of ranking and contrastive terms over both clip saliency scores and sentence-level attention weights, and a phrase-level loss with two parts: a Distinct Query Attention loss that pushes phrase attention distributions toward orthogonality using a Frobenius-norm penalty, and an [EOS] reconstruction loss (InfoNCE with a temperature parameter) that aligns the reconstructed phrase token P_[EOS] with the original [EOS] embedding. Features are extracted without fine-tuning.

Why This Matters

The paper challenges a default assumption in multimodal grounding — that a pretrained text encoder's [EOS] token is the right thing to attend to — and shows that explicitly preserving word-level structure yields measurable gains, especially at strict IoU thresholds and for long queries. This points toward treating different tokens according to their semantic role rather than uniformly, an idea that transfers to other video-language tasks.

Real-world applications:

• Video search and retrieval: Finding the exact moment in a long recording that matches a free-text description, where long or detailed queries are common.
• Media editing and highlight reels: Automatically scoring and extracting salient clips from sports footage, lectures, or event recordings, which is the Highlight Detection task.
• Surveillance and body-cam review: Localizing specific described events within hours of untrimmed footage using queries that hinge on concrete visual details.
• Assistive and accessibility tools: Letting users jump to a described moment in a video by text query rather than manual scrubbing.

Industry relevance: the author list includes an affiliation with LG Electronics, and the evaluated benchmarks are built on consumer video platforms (open-domain YouTube content via QVHighlights, plus Charades-STA and TVSum), suggesting direct applicability to video platforms, content recommendation, and automated media production pipelines. The gains over strong InternVideo2-based baselines indicate the improvement comes from the architecture rather than feature quality, which matters for teams that already have fixed pretrained encoders.

Future Directions

1. Adaptive phrase segmentation: The model assumes a fixed number of phrases per query, which currently requires manual adjustment per dataset. Deriving the phrase count from query structure is the authors' stated direction.
2. Audio-visual grounding: The model does not use audio features, which the authors note may limit performance in audio-visual grounding scenarios.
3. Long-query robustness as a design target: The advantage over baselines is largest for queries over 20 words (48.92 mAP versus 43.46 for the next best), raising the question of how far that scaling can be pushed and whether phrase structure alone accounts for it.
4. Reducing the supervision burden: Phrase quality depends on the DQA and [EOS] reconstruction losses, and the ablation shows HD metrics can degrade without them; a better understanding of how to regulate the phrase path without hurting saliency estimation is left open.

Target Audience

Researchers and engineers working on video-language models, video moment retrieval, and highlight detection; practitioners building text-based video search or automatic editing systems; and readers interested in how token-level semantic roles affect cross-modal attention. The paper is most useful to those already comfortable with VTG baselines such as FlashVTG, CG-DETR, and TR-DETR, since much of its argument is framed as a comparison against them.

Authors’ abstract

Video Temporal Grounding (VTG) aims to localize temporal segments in long, untrimmed videos that align with a given natural language query. This task typically comprises two subtasks: Moment Retrieval (MR) and Highlight Detection (HD). While recent advances have been progressed by powerful pretrained vision-language models such as CLIP and InternVideo2, existing approaches commonly treat all text tokens uniformly during crossmodal attention, disregarding their distinct semantic roles. To validate the limitations of this approach, we conduct controlled experiments demonstrating that VTG models overly rely on [EOS]-driven global semantics while failing to effectively utilize word-level signals, which limits their ability to achieve fine-grained temporal alignment. Motivated by this limitation, we propose DualGround, a dual-branch architecture that explicitly separates global and local semantics by routing the [EOS] token through a sentence-level path and clustering word tokens into phrase-level units for localized grounding. Our method introduces (1) tokenrole- aware cross modal interaction strategies that align video features with sentence-level and phrase-level semantics in a structurally disentangled manner, and (2) a joint modeling framework that not only improves global sentence-level alignment but also enhances finegrained temporal grounding by leveraging structured phrase-aware context. This design allows the model to capture both coarse and localized semantics, enabling more expressive and context-aware video grounding. DualGround achieves state-of-the-art performance on both Moment Retrieval and Highlight Detection tasks across QVHighlights and Charades- STA benchmarks, demonstrating the effectiveness of disentangled semantic modeling in video-language alignment.