Mitigating Semantic Collapse in Partially Relevant Video Retrieval

AI summary

Overview

Research area: Computer vision, specifically cross-modal (text-to-video) retrieval and multimodality representation learning.

Technical level: Advanced. Familiarity with contrastive learning (InfoNCE), CLIP-style joint embedding spaces, transformer tokenization, and retrieval benchmarks is assumed.

Scope: This paper diagnoses and mitigates "semantic collapse" in Partially Relevant Video Retrieval (PRVR) by introducing a text-side distillation objective and a video-side contrastive alignment method, validated across four retrieval benchmarks.

What This Paper Is About

In Partially Relevant Video Retrieval, a system must retrieve an untrimmed video when only some of its content matches a text query. The paper identifies a critical flaw: standard training treats every text query paired with a given video as a positive and everything else as negative, which forces semantically unrelated queries from the same video to cluster together, while pulling apart semantically similar queries from different videos. The same distortion happens on the video side, where temporally distinct segments from one video collapse into nearly identical embeddings. The goal is to prevent this collapse in both the text and video embedding spaces.

Key Contributions

1. Text Correlation Preservation Learning (TCPL): A distillation objective that transfers the pairwise Euclidean and triplet angular relationships among text queries from CLIP's well-structured semantic space into the PRVR retrieval space, preserving meaningful query-to-query structure rather than indiscriminately repelling queries from the same video.

2. Cross-Branch Video Alignment (CBVA): A contrastive objective leveraging the standard dual-branch (frame-level and clip-level) architecture. It aligns a frame token with its timestamp-matched clip token and repels it from clip tokens at other timestamps, explicitly disentangling distinct events within a single video.

3. Order-Preserving Token Merging (OP-ToMe) and Adaptive CBVA: OP-ToMe restricts merging to temporally adjacent frame tokens, producing clip segments that are internally coherent and mutually distinctive. Adaptive CBVA estimates how many genuinely distinct contexts a video contains and merges redundant clips accordingly.

4. State-of-the-art results on four PRVR benchmarks (QVHighlights, TVR, ActivityNet Captions, Charades-STA), with the largest margins on datasets where semantic collapse is most severe.

Main Findings

• Both branches of collapse need fixing: Component-wise ablation on QVHighlights shows SumR rising from 225.5 (baseline) to 230.6 (+TCPL), then to 231.0 (naive CBVA), 232.5 (+OP-ToMe), and 234.6 (+Adaptive CBVA), confirming that text-side and video-side remedies are complementary.

• Naive CBVA alone is weak: Simply aligning clip and frame branches yields only marginal gains because uniform average-pooled clip segments span overlapping, mixed contexts. OP-ToMe and adaptive merging are what make CBVA effective.

• Video-side distillation (VCPL) underperforms CBVA: Distilling CLIP's relationships directly onto video embeddings trails CBVA by 2.3 SumR points, because CLIP's image-pretrained space lacks temporal modeling capacity.

• Better source model, better student: Swapping CLIP-B for CLIP-L or OpenCLIP-L as the TCPL teacher raises SumR from 234.6 to 235.6 and 236.4 respectively, showing the method scales with teacher quality.

• Loss weighting favors angular over Euclidean relationships: A 1:2 ratio of Euclidean to angular loss (λE=15, λA=30) is optimal, suggesting triplet angular structure carries more useful signal for preventing collapse.

• Robust to hyperparameters: The similarity threshold τ used to estimate video context count shows only minor performance fluctuation between 0.5 and 0.8, and a 75% merge rate clearly beats 50%.

• Leaderboard results: QVHighlights SumR 234.6 (vs. 225.2 for GMMFormer-v2), TVR 263.1 (vs. 260.1 for ARL), ActivityNet Captions 202.1 (vs. 196.9 for GMMFormer-v2), Charades-STA 99.7 (vs. 96.7 for BGM-Net).

• Semantic structure is measurably preserved: Spearman rank correlation with CLIP's query embeddings jumps from 35.40 (baseline) to 68.18, and the normalized intra-video versus cross-video similarity gap drops from 0.82 to 0.46, indicating deliberate relaxation of over-clustering rather than mere separation enforcement.

• Inference efficiency is preserved: Inference time and memory are the second-lowest among compared methods and scale better than MS-SL as database size grows. Training is significantly heavier (62,641 ms/epoch vs. 17,223 for GMMFormer-v2), but the paper argues this cost is paid offline while inference governs deployment.

Methodology in Plain English

The system uses pretrained encoders (CLIP for text and video, plus Slowfast in some configurations) to extract tokens, then passes them through trainable transformer layers. Two parallel branches encode video at different temporal granularities: one for individual frames, one for multi-frame clips.

To fix the text side: The researchers take the text embeddings CLIP already produces and measure two things among queries in a training batch — how far apart they are (Euclidean distance) and what angle three of them form (angular distance). They then train the model to reproduce those same distance and angle patterns in its own embedding space using a Huber loss, which behaves like squared error for small mistakes and absolute error for large ones. This means the model learns CLIP's nuanced notion of which queries are actually related, instead of just assuming all queries tied to the same video are equivalent.

To fix the video side: The model is trained so that a frame at a given timestamp is close to the clip covering that timestamp, and far from clips covering other timestamps in the same video. This forces the model to distinguish, say, a cooking segment from a skiing segment within one video, rather than letting both collapse into the same representation.

Making clips coherent: Instead of blindly averaging fixed-length windows, the method merges only adjacent frames in temporal order, repeatedly, until 128 frames become 32 clips. New tokens representing merged content are created via size-weighted averaging, so the model still knows how many original frames each clip covers.

Adapting to video content: A uniform 32-clip split is wasteful for a video with only one or two distinct events. The method measures pairwise similarity among a video's clips, computes what fraction exceed a threshold, and if enough clips are similar, merges further using bipartite matching until the clip count plausibly reflects the number of genuinely distinct contexts. A discrete ladder of candidate clip counts is precomputed to avoid expensive per-video searches.

Why This Matters

Impact on research: This paper reframes PRVR training as a problem of preserving semantic structure rather than enforcing separation. Prior work (GMMFormer, GMMFormer-v2) pushed apart queries from the same video without regard for whether they were actually related. This paper shows that distilling relational knowledge from a foundation model preserves appropriate coherence while still preventing collapse, and it is the first to systematically address collapse on the video-embedding side, not just the text side. The finding that CLIP's image-based semantics cannot simply be transferred to video makes a case for modality-specific alignment objectives.

Real-world applications:

• Video search engines that must surface a long documentary or livestream because one minute of it addresses the user's query.
• Media and news archives, where a single untrimmed broadcast contains many stories and a text query targets only one.
• Content recommendation and clip-finding systems that need to locate a precise moment inside long user-generated video.
• Surveillance and legal review, where an analyst queries untrimmed footage for a specific activity that occurs for only a short interval.
• Educational platforms indexing long lecture recordings against topic-level queries.

Industry relevance: Any platform that stores untrimmed video and serves text-based discovery — streaming services, video-sharing sites, enterprise media asset managers, and enterprise search vendors — benefits directly. The dual constraint of negligible inference overhead (features are precomputed) and no increase in model parameters means the method can be dropped into existing retrieval pipelines without retraining costs at query time.

Future Directions

1. Reducing training cost. The near-4x increase in per-epoch training time versus GMMFormer-v2 (62,641 ms vs. 17,223 ms) is the paper's main practical weakness. Distilling relations more cheaply, or approximating TCPL with cached teacher statistics, is a natural follow-up.

2. Richer context estimation. Adaptive CBVA currently estimates the number of distinct video contexts via a similarity threshold and a bipartite merge ladder. Learning this count end-to-end, or inferring it from the text query rather than the video alone, could improve precision.

3. Beyond CLIP as teacher. The ablation shows performance scales with teacher quality (OpenCLIP-L gives the best text-side result). Investigating larger, more video-aware teachers — or multimodal LLMs — for relation distillation is an open opportunity.

4. Cross-modal and temporal grounding extensions. Since CBVA explicitly models temporal correspondence, extending the framework to moment-level temporal grounding, or to audio-visual retrieval, is a logical next step.

5. Domain transfer and low-resource settings. All results are on four English-language, Western-centric benchmark datasets. Whether semantic collapse behaves similarly in specialized domains (medical video, industrial inspection, non-English content) remains untested.

Target Audience

Multimodal retrieval and video-language researchers who are already familiar with contrastive learning and dual-encoder architectures will gain the most, especially those building on GMMFormer or ProtoPRVR. It is also valuable for machine learning engineers implementing video search systems who need an efficient, drop-in objective that preserves inference speed — though the increased training budget should be weighed. Readers with only a beginner-level background will find the loss formulations and token-merging mechanics challenging without prior exposure to InfoNCE and CLIP.

Authors’ abstract

Partially Relevant Video Retrieval (PRVR) seeks videos where only part of the content matches a text query. Existing methods treat every annotated text-video pair as a positive and all others as negatives, ignoring the rich semantic variation both within a single video and across different videos. Consequently, embeddings of both queries and their corresponding video-clip segments for distinct events within the same video collapse together, while embeddings of semantically similar queries and segments from different videos are driven apart. This limits retrieval performance when videos contain multiple, diverse events. This paper addresses the aforementioned problems, termed as semantic collapse, in both the text and video embedding spaces. We first introduce Text Correlation Preservation Learning, which preserves the semantic relationships encoded by the foundation model across text queries. To address collapse in video embeddings, we propose Cross-Branch Video Alignment (CBVA), a contrastive alignment method that disentangles hierarchical video representations across temporal scales. Subsequently, we introduce order-preserving token merging and adaptive CBVA to enhance alignment by producing video segments that are internally coherent yet mutually distinctive. Extensive experiments on PRVR benchmarks demonstrate that our framework effectively prevents semantic collapse and substantially improves retrieval accuracy.