Unified Interactive Multimodal Moment Retrieval via Cascaded Embedding-Reranking and Temporal-Aware Score Fusion

AI summary

Overview

Research area: Multimodal video retrieval and interactive moment search (computer vision, cross-modal retrieval, video browsing).

Technical level: Advanced — the paper combines several large pretrained models (BEiT-3, SigLIP, BLIP-2, GPT-4o, Whisper Large-v3, Gemini 2.0 Flash) with beam search, exponential temporal decay, and score-fusion mathematics.

Scope: The paper describes a complete multimodal video search system — an offline indexing pipeline over video, OCR, and speech, plus an online retrieval pipeline driven by an LLM agent — and reports its results in the AI Challenge 2025 competition.

What This Paper Is About

Users searching video collections rarely say which channel to search — the visuals, the on-screen text, or the spoken audio — and they often query from vaguely remembered details. The paper argues that fixed-weight fusion of modalities breaks down under noisy audio and OCR errors, that temporal models struggle to return coherent event sequences instead of isolated frames, and that forcing users to pick a modality hurts usability. The goal is a single system that takes a free-form query, figures out which modalities matter and how much, retrieves candidates, and assembles them into a temporally plausible sequence of moments.

Key Contributions

1. Cascaded dual-embedding retrieval pipeline. BEiT-3 and SigLIP embeddings are combined for broad first-pass retrieval over keyframes, and the candidates are refined by a BLIP-2 image-text-matching reranker, trading off recall in stage one against precision in stage two.
2. Temporal event reasoning with exponential decay. A beam-search procedure constructs coherent event sequences and applies an exponential decay factor (λᵢ = e^(−α·Δtᵢ)) that softly penalizes large time gaps between consecutive events instead of using hard cutoffs.
3. Agent-guided multimodal query decomposition and fusion. GPT-4o interprets ambiguous queries, splits them into visual, OCR, and ASR sub-queries, assigns per-modality weights, and drives an adaptive score fusion that removes the need for manual modality selection.
4. Query expansion module. A GPT-4o-based Query Expansion module generates N = 4 query variants (default), where the first is always a direct English translation of the original and the rest vary visual angle, setting, or description style without introducing new objects or actions.

Main Findings

• Competition score: The system achieved a final score of 76.4/88 (86.8%) on the AI Challenge 2025 dataset, which the authors report placed it among the top-performing teams and advanced it to the final round.
• Per-round scores: Round 1 scored 19.8/23 (86.1%), Round 2 scored 26.6/30 (88.6%), and Round 3 scored 30/35 (85.7%), the maximum score in Round 3.
• Query mix: Across all rounds the evaluation contained 72 KIS queries, 9 VQA queries, and 7 TRAKE queries. Round 1 had 17 KIS / 3 VQA / 3 TRAKE; Round 2 had 26 / 2 / 2; Round 3 had 29 / 4 / 2 — the highest KIS proportion in Round 3.
• Reranking is decisive: Qualitative analysis shows the broad-search stage alone ("Without Rerank") retrieved a wide variety of candidates but missed the ground truth entirely, while the BLIP-2 second-stage reranker pushed the correct frame to the top position.
• Adaptive fusion beats visual-only retrieval: In the reported example, the Agent guided the system to weight OCR higher (w_ocr ≈ 0.7) than visual features (w_vis ≈ 0.4), allowing it to pick the correct scene from dozens of near-identical visuals by prioritizing the on-screen text on a sign.
• Temporal decay prefers compact sequences: The system returned a three-event factory assembly sequence with a duration of 5.1 seconds and a score of 0.9234, using λ = 0.010 (decay coefficient 0.01, beam width 8), while penalizing a competing candidate that spanned 34.1 seconds and scored lower.
• Evaluation is qualitative for the design choices: The paper reports no numeric ablation study comparing the cascaded pipeline, the decay term, or the fusion strategy against alternatives — the evidence for these components is presented through figure-based examples rather than measured metrics. No runtime or latency figures are reported.

Methodology in Plain English

The system is split into an offline half and an online half.

Offline, the audio track is separated from the video, and TransNetV2 detects shot boundaries. Three representative keyframes are extracted per shot. Each keyframe is turned into two normalized embeddings — one from BEiT-3, one from SigLIP — and both are stored together in the Qdrant vector database using named vectors. In parallel, on-screen text is extracted from each keyframe by Gemini 2.0 Flash using a JSON-based prompt, and speech is transcribed by Whisper Large-v3 into timestamped segments. Speech segments are aligned to the nearest keyframes so that a spoken phrase maps back to a visual segment.

Online, a GPT-4o agent reads the query and splits it into visual, OCR, and ASR components with weights attached to each, based on prompt-embedded heuristics (KIS = things you see, OCR = things you read, ASR = things you hear). The expanded query variants (N = 4) are embedded and searched in parallel across three branches: a visual branch in Qdrant using cosine matching with Score-Reflected Reciprocal Rank Fusion over the two embedding models, keeping the top 100 candidates for BLIP-2 reranking; and OCR/ASR branches in Elasticsearch using exact phrase, full-term, partial, and fuzzy matching. Scores from each modality are min-max normalized (Equation 1) and then combined with the agent-predicted weights (Equation 2). For multi-event queries, a beam search keeps only the top-B partial sequences, reducing the alignment cost from exponential to O(B × K × M) and avoiding combinatorial blow-up. Each event transition is weighted by an exponential decay term, and the sequence score is built additively so that one weak transition does not invalidate the whole chain (Equations 4 and 5). Finally, the best sequence goes through a BLIP-2 validation stage where the final per-event score is the product sᵢ · λᵢ · bᵢ — a multiplicative gate that suppresses results that fail any one of semantic relevance, temporal coherence, or fine-grained alignment (Equations 6 and 7).

Why This Matters

Impact on research. The paper is a concrete engineering answer to a known gap: most retrieval systems are single-stage, fixed-weight, and temporally naive. It shows how to combine dual encoders, a cross-encoder reranker, LLM-driven query decomposition, and a sequence-level temporal prior within one deployed pipeline, and it provides an alternative to hard temporal thresholds and to local stability measures such as ABTS by enforcing a global decay constraint across the whole sequence.

Real-world applications:

• Searching large broadcast, news, or documentary archives where the answer might appear as a caption on screen, a spoken line, or a visual scene.
• Sports footage retrieval, where the visual action, a jersey name read on screen, and commentary all carry different amounts of signal.
• Enterprise media asset management, where editors need specific moments out of terabytes of footage without tagging every clip by hand.
• Educational and tutorial video search, where a lecture's spoken explanation, its slides, and the demonstrated action are separate modalities.

Industry relevance. The system is built entirely from off-the-shelf components (Qdrant, Elasticsearch, TransNetV2, Whisper, Gemini, GPT-4o, BEiT-3, SigLIP, BLIP-2), which makes it directly reproducible or adaptable by teams operating video platforms. The competition result of 76.4/88 on the AI Challenge 2025 dataset, comprising almost 1,500 videos and more than 200 GB of multimodal data, is evidence that the architecture holds up at realistic scale.

Future Directions

1. Semantic temporal modeling. The authors plan to move beyond exponential decay alone and add models that understand semantic relationships between events, so the system produces meaningful storylines rather than frames that merely happen to be close in time.
2. Unified multimodal captioning. They intend to add a multimodal language model that processes both images and audio to generate a single caption per video segment, then embed those captions — replacing the current practice of indexing each modality separately.
3. Better interface and user feedback. The plan is a more intuitive user interface plus a feedback loop where the system adjusts its fusion strategies over time based on user input.
4. Benchmark evaluation. The conclusion calls for large-scale evaluation on TRECVID and VBS in addition to the reported AI Challenge 2025 results.

Target Audience

Competition participants and practitioners building multimodal video search systems; researchers in video retrieval, VBS/TRECVID-style interactive search, and multimodal fusion; and engineers who need a practical blueprint for combining dual encoders, cross-encoder reranking, LLM query routing, and temporal scoring into a working pipeline. Readers looking for a controlled ablation study or rigorous statistical comparison of the three proposed components will not find one here — the quantitative evidence is a competition score, and the component-level evidence is qualitative.

Authors’ abstract

The exponential growth of video content has created an urgent need for efficient multimodal moment retrieval systems. However, existing approaches face three critical challenges: (1) fixed-weight fusion strategies fail across cross modal noise and ambiguous queries, (2) temporal modeling struggles to capture coherent event sequences while penalizing unrealistic gaps, and (3) systems require manual modality selection, reducing usability. We propose a unified multimodal moment retrieval system with three key innovations. First, a cascaded dual-embedding pipeline combines BEIT-3 and SigLIP for broad retrieval, refined by BLIP-2 based reranking to balance recall and precision. Second, a temporal-aware scoring mechanism applies exponential decay penalties to large temporal gaps via beam search, constructing coherent event sequences rather than isolated frames. Third, Agent-guided query decomposition (GPT-4o) automatically interprets ambiguous queries, decomposes them into modality specific sub-queries (visual/OCR/ASR), and performs adaptive score fusion eliminating manual modality selection. Qualitative analysis demonstrates that our system effectively handles ambiguous queries, retrieves temporally coherent sequences, and dynamically adapts fusion strategies, advancing interactive moment search capabilities.