When One Moment Isn't Enough: Multi-Moment Retrieval with Cross-Moment Interactions

AI summary

Overview

Research area: Computer vision / video-language understanding, specifically video temporal grounding and moment retrieval.

Technical level: Intermediate. The paper assumes familiarity with moment retrieval (MR), object-detection-style proposals, Transformer encoders, and standard metrics such as mAP and IoU, but explains its new dataset and metrics in accessible terms.

Scope: The paper introduces a multi-moment retrieval (MMR) benchmark dataset built on top of QVHighlights, a matching set of evaluation metrics, and a new model called FlashMMR, then benchmarks it against 6 existing MR methods.

What This Paper Is About

Almost all existing moment retrieval work assumes a query matches exactly one moment in a video (Single-Moment Retrieval, or SMR), but in real videos one query often matches several non-overlapping moments — for example, "cutting vegetables" recurring across an instructional video, or "successful three-point shots" occurring repeatedly in a sports broadcast. The paper's goal is to close this gap by providing a dataset and metrics for Multi-Moment Retrieval (MMR) and a model designed to retrieve all relevant moments rather than just the single best one.

Key Contributions

1. QV-M² (QVHighlights Multi-Moment Dataset): Described as the first fully human-annotated dataset dedicated to MMR benchmarking, containing 2,212 new query annotations and 6,384 annotated temporal moments across 1,341 videos, averaging 2.9 moments per query.
2. New MMR evaluation metrics: Generalized mAP (G-mAP), mAP broken down by number of ground-truth moments (mAP@1_tgt, mAP@2_tgt, mAP@3+_tgt), mean IoU@k, and mean Recall@k, extending standard SMR protocols to one-to-many retrieval.
3. FlashMMR, a dedicated MMR framework: Built around a Multi-Moment Post-Verification module that performs constrained temporal adjustment and semantic-consistency-based re-evaluation to prune low-confidence proposals.
4. A systematic benchmark: 6 existing MR methods are retrained and evaluated on QV-M² and QVHighlights under both SMR and MMR settings, establishing baselines and showing where SMR architectures break down.

Main Findings

• FlashMMR sets a new state of the art on QV-M²: On the QV-M² test set, FlashMMR reaches a G-mAP of 35.14 versus 32.14 for FlashVTG, with 52.59 on mAP@1_tgt, 42.52 on mAP@2_tgt, and 22.89 on mAP@3+tgt.
• Reported improvements over the prior SOTA: The paper reports gains of 3.00% on G-mAP, 2.70% on mAP@3+tgt, and 2.56% on mR@3 on QV-M² relative to the previous best method.
• QV-M² improves training even for SMR-oriented models: Models trained on QV-M² consistently outperform counterparts trained only on QVHighlights. For example, FlashVTG improves from 48.02 to 48.35 G-mAP in the cross-dataset comparison, and M-DETR from 32.79 to 34.70.
• Evaluation on QV-M² is harder for existing methods: All methods show a performance drop when QV-M² is used for evaluation rather than training, which the authors attribute to the increased number of one-to-many moment queries.
• Existing SMR methods transfer poorly to MMR: Performance on mAP@3+tgt is much lower across the board than on mAP@1_tgt, for instance M-DETR at 10.95 versus 33.71 on QV-M², illustrating the architectural difficulty of dense multi-moment localization.
• The Post-Verification module is consistently beneficial: The ablation shows QV-M² G-mAP rising from 32.14 without PV to 35.14 with PV, mAP@2_tgt from 39.48 to 42.52, mAP@3+tgt from 20.19 to 22.89, mIoU@2 from 47.85 to 49.64, mIoU@3 from 40.92 to 42.92, mR@2 from 41.30 to 44.33, and mR@3 from 35.94 to 38.50. On the easier QVHighlights set, gains are smaller but still present (G-mAP 48.02 to 48.07, mAP@2_tgt 35.08 to 35.78).
• QV-M² is denser than prior MR datasets: Its average of 2.9 moments per query exceeds QVHighlights (1.8) and NExT-VMR (1.5), while other listed datasets such as DiDeMo, ANetCaptions, CharadesSTA, TVR, TACoS, YouCook2, COIN and HiREST average 1.
• Annotation detail: Most annotated temporal windows fall within the 2 to 20-second range, with 1,263 instances (19.8%) extending beyond 20 seconds; QV-M² moments are distributed more uniformly across videos than Charades-STA's.
• Metrics maintain SMR compatibility: G-mAP, mIoU@1, and mR@1 remain fully consistent with standard SMR metrics, and mIoU@k/mR@k are computed only on queries with at least k ground-truth moments.

Methodology in Plain English

Dataset construction. The authors keep the original QVHighlights videos (YouTube lifestyle vlogs and news footage, 5 to 30 minutes long, from first-person and third-person perspectives) and add new annotations for the multi-moment setting. Annotators follow guidelines to write detailed queries capturing actors, actions and context; context-dependent queries requiring knowledge of temporal relationships; and "negative" (inverse) queries marking segments where an action does not occur. Each query is linked to one or more start/end segments. For quality control, after every 100 videos are annotated, 5% are randomly re-checked by a second annotator; if the two annotators' temporal boundaries overlap by less than 90%, the whole batch is re-annotated by a third annotator.

Metrics. A prediction counts as a true positive if its IoU with any unmatched ground-truth moment passes a threshold (e.g., 0.5). G-mAP averages AP over several IoU thresholds (e.g., 0.5, 0.55, ..., 0.9). mAP is additionally broken down by how many ground-truth moments a query has. Mean IoU@k averages the best IoU achieved by the top-k predictions, and mean Recall@k checks whether each ground-truth moment is matched by any of the top-k predictions.

FlashMMR model. Video features come from frozen SlowFast and CLIP encoders at 0.5 FPS, and text features from CLIP; both are projected into a shared space. A cross-attention module with learnable dummy tokens fuses video and text, and a Transformer encoder refines the result for long-range dependencies. A temporal feature pyramid built from strided 1D convolutions captures moments at different durations, with boundary predictions and confidence scores produced at each scale. Confidence is combined from intra-scale and inter-scale scores using a learnable weighting factor.

Post-verification. The model first produces 3 candidate moments per query. A post-processing function enforces minimum/maximum window lengths, clipping, and rounding; the refined intervals are then used to slice the fused features. A GRU-based post-verification network re-scores each interval, supervised against the temporal IoU between predictions and ground-truth moments, plus a representation loss that encourages semantically similar segments to have consistent similarity patterns. Training combines Focal Loss, L1 Loss, Clip-Aware Score Loss, and the post-verification loss (weighted 9 for the verification term and 7 for the representation term), using AdamW. Inference uses an NMS threshold of 0.7, and all experiments run on a single RTX 4090 GPU. QVHighlights SMR verification experiments use the validation set because test-set annotations are unavailable.

Why This Matters

Impact on research. The paper reframes video temporal grounding around a realistic assumption — that a query can map to many moments — and supplies both a benchmark and a baseline. It shows that architectures designed for single-moment retrieval degrade in dense one-to-many settings, which makes MMR a distinct research problem rather than a minor variation of SMR.

Real-world applications (the first two are examples given in the paper):

• Instructional videos, where a query like "cutting vegetables" corresponds to many separate chopping instances of different ingredients.
• Sports broadcasts, where a query like "successful three-point shots" occurs multiple times in a single match.
• Video search and content management, where users want every relevant clip from a long recording rather than one ranked result.
• Video editing and summarization workflows that need a complete set of relevant segments to assemble or index footage.

Industry relevance. Any system that indexes or searches long-form video — media archives, e-learning platforms, sports analytics, and content moderation tooling — faces the one-to-many mismatch this paper targets. The dataset and metrics give those systems a way to measure whether they retrieve all relevant moments, not just the most likely one.

Future Directions

• Stronger verification and discrimination. The authors state their verification module is at an early stage and suggest reinforcement learning or contrastive learning to better distinguish moments.
• Scaling annotated data. They flag the limited size of high-quality MMR annotations as a constraint on future progress, since QV-M² may be insufficient as models become more advanced.
• Closing the metric gap on dense queries. mAP@3+tgt remains far below mAP@1_tgt for every method, so architectures that explicitly model dependencies among multiple moments are an open problem.
• Broader task coverage. The related work notes that concurrent work on NExT-VMR extends retrieval to multi-moment and no-moment cases; combining those settings with standardized, publicly available annotations remains an open direction (the paper notes NExT-VMR is not yet publicly available).

Target Audience

Researchers and graduate students in computer vision and video-language understanding, particularly those working on video temporal grounding, moment retrieval, and multimodal retrieval. It is also useful for practitioners building video search, indexing, or editing systems who need to know why single-moment assumptions fail and how to measure multi-moment performance. Readers should be comfortable with standard detection-style metrics (mAP, IoU, recall) and Transformer-based architectures.

Authors’ abstract

Existing Moment retrieval (MR) methods focus on Single-Moment Retrieval (SMR). However, one query can correspond to multiple relevant moments in real-world applications. This makes the existing datasets and methods insufficient for video temporal grounding. By revisiting the gap between current MR tasks and real-world applications, we introduce a high-quality datasets called QVHighlights Multi-Moment Dataset (QV-M$^2$), along with new evaluation metrics tailored for multi-moment retrieval (MMR). QV-M$^2$ consists of 2,212 annotations covering 6,384 video segments. Building on existing efforts in MMR, we propose a framework called FlashMMR. Specifically, we propose a Multi-moment Post-verification module to refine the moment boundaries. We introduce constrained temporal adjustment and subsequently leverage a verification module to re-evaluate the candidate segments. Through this sophisticated filtering pipeline, low-confidence proposals are pruned, and robust multi-moment alignment is achieved. We retrain and evaluate 6 existing MR methods on QV-M$^2$ and QVHighlights under both SMR and MMR settings. Results show that QV-M$^2$ serves as an effective benchmark for training and evaluating MMR models, while FlashMMR provides a strong baseline. Specifically, on QV-M$^2$, it achieves improvements over prior SOTA method by 3.00% on G-mAP, 2.70% on mAP@3+tgt, and 2.56% on mR@3. The proposed benchmark and method establish a foundation for advancing research in more realistic and challenging video temporal grounding scenarios. Code is released at https://github.com/Zhuo-Cao/QV-M2.