MARS: What Retrieval Signals Are Hidden in Multimodal Large Language Models for Text-Video Retrieval?

AI summary

Overview

• Research area: Computer vision / cross-modal retrieval — specifically text-video retrieval using multimodal large language models (MLLMs) as embedding extractors.
• Technical level: Advanced. The paper assumes familiarity with contrastive representation learning (InfoNCE), transformer decoder layers, LoRA fine-tuning, and standard retrieval metrics (Recall@K).
• Scope: The paper proposes MARS, a multi-layer, multi-slot embedding framework that replaces single-token, final-layer embedding extraction in MLLM-based text-video retrieval, and evaluates it on four retrieval benchmarks plus text-image retrieval.

What This Paper Is About

Most recent methods that adapt multimodal large language models into retrieval embedders represent an entire input with one vector taken from one token in the final decoder layer. For video, which contains objects, actions, and temporal dynamics, this creates an information bottleneck that hurts fine-grained retrieval, especially among visually similar candidates. MARS instead builds several adaptive representation slots by fusing hidden states from multiple decoder layers, matches text and video slots one-to-one, and adds a hard-negative-aware specialization objective so the slots learn to separate confusing negatives.

Key Contributions

1. Reframing embedding extraction for MLLM-based text-video retrieval. The authors revisit the standard "one token, one layer" extraction and propose MARS (Multi-layer Adaptive Representation Slots), which constructs adaptive representation slots by fusing layer-wise hidden states with slot-specific learned layer weights.
2. A slot-based matching and training scheme. MARS compares corresponding text and video slots and aggregates their similarities, and introduces both a hard-negative-aware slot specialization loss and a slot diversity loss to prevent redundancy and encourage discriminative slots.
3. State-of-the-art results under a dual-encoder structure. Experiments on four text-video retrieval benchmarks (DiDeMo, ActivityNet, LSMDC, MSR-VTT) show the best results in both direct similarity-based retrieval and reranking settings, while keeping an efficient dual-encoder pipeline.
4. In-depth analysis of why it works. The paper dissects layer-wise retrieval evidence, slot complementarity, slot distinctness, computational cost, and generalization across MLLM backbones and to text-image retrieval.

Main Findings

• Best direct similarity-based retrieval. MARS reaches 67.0 T2V mR@1 and 64.5 V2T mR@1 averaged across the four benchmarks, the strongest among direct similarity-based retrieval methods, outperforming InternVideo2-6B* even though that method relies on cross-modal matching.
• Score calibration helps further. With DSL post-processing, "MARS*" improves to 71.6 T2V mR@1 and 72.0 V2T mR@1.
• Best reranking result. MARS-R, which retrieves with MARS and reranks with BLiM, achieves the best T2V mR@1 of 73.2 and 67.9 V2T mR@1, improving over BLiM (72.0 and 67.1) in both average T2V and V2T performance.
• Multi-layer fusion gives the largest single ablation gain. Ablation shows average R@1 across the eight T2V/V2T settings rising from 61.2 (final-layer, single-slot baseline) to 63.3 with layer fusion, to 65.1 after adding slots, then to 65.7 after adding diversity regularization and hard-negative-aware slot specialization.
• A moderate number of slots is optimal. Average R@1 is 75.6 at M=2, 76.3 at M=3, peaks at 77.7 with M=4, and drops to 77.4 at M=5 and 75.9 at M=6.
• Slot aggregation beats alternatives. Uniform aggregation achieves an average R@1 of 65.7, compared with 65.4 for learnable aggregation and 65.1 for MaxSim.
• Retrieval evidence is spread across decoder layers. In the final-layer baseline, R@1 rises toward the final layer; with multi-layer fusion trained, several upper decoder layers achieve comparable R@1, and some layers with lower individual R@1 still receive non-negligible fusion weights.
• Slots are complementary. Aggregation outperforms the strongest individual slot by +12.3 R@1 on DiDeMo, +10.2 on ActivityNet, +11.5 on LSMDC, and +9.6 on MSR-VTT. Slot-exclusive rank-1 cases (where only one slot retrieves the correct video at rank one) number 123 on DiDeMo, 785 on ActivityNet, 191 on LSMDC, and 209 on MSR-VTT.
• Slots stay distinct rather than collapsing. Mean off-diagonal top-1 agreement is 0.537 on DiDeMo, 0.429 on ActivityNet, 0.206 on LSMDC, and 0.419 on MSR-VTT, and pairwise cosine similarities between video-side slot embeddings remain generally small.
• Low online cost. MARS has the lowest online cost across all four benchmarks, with per-query online costs ranging from 274.3 to 1,031.8 GF/query versus UMT's 1,395.2 to 2,683.4 GF/query. Total computation is 51.1 M GFLOPs for MARS, compared with 61.7 for InternVideo2-1B and 134.0 for InternVideo2-6B, and evaluation time is 3,626.1s versus 8,717.4s and 10,424.3s respectively.
• Generalizes across backbones. Against their respective single-representation baselines, MARS improves average R@1 by +3.0 for VideoLLaMA3-7B, +3.1 for Qwen2-VL-7B, and +2.3 for Qwen2-VL-2B (the text states consistent absolute improvements ranging from 2.0% to 3.4% across the three backbones).
• Generalizes to text-image retrieval. On COCO (fine-tuned), MARS reaches 66.3 T2I and 83.9 I2T R@1, 1.2 points higher than BEiT-3 in both directions; on Flickr30K (zero-shot) it reaches 89.1 T2I and 97.0 I2T R@1, while BLIP-2 reports higher scores using an additional cross-modal reranking stage.

Methodology in Plain English

The base model is VideoChat-Flash-Qwen2-7B, which pairs a UMT-L vision encoder with a 28-layer Qwen2 language model. Instead of asking the model to summarize an input in a single token, the authors place several learnable "adaptive representation tokens" in the assistant response field of the prompt (between 3 and 4 tokens, depending on the dataset). Because the model uses causal attention, the hidden state immediately before each token serves as that slot's representation.

Each slot then combines hidden states from all decoder layers using its own learned set of layer weights, so different slots can emphasize different depths of the network. Text and video slots are L2-normalized and compared slot-by-slot with cosine similarity, and the slot similarities are averaged into a single retrieval score. Training uses three objectives: a symmetric InfoNCE contrastive loss, a diversity loss that penalizes high similarity between slots from the same input, and a hard-negative-aware slot specialization loss that selects the hardest negative for each query and requires at least one slot to separate the positive from it by a margin of 0.04. Only the adaptive representation tokens, the layer-fusion weights, and LoRA parameters are trained; all pretrained weights stay frozen. Videos are sampled at 16 frames, training uses two NVIDIA A100 40GB GPUs with gradient checkpointing, and the datasets are DiDeMo, ActivityNet Captions, LSMDC, and MSR-VTT.

Why This Matters

• Impact on research: The paper challenges the default assumption that MLLM embeddings should come from a single final-layer token, and shows that retrieval evidence is distributed across decoder depth. It also reframes hard negatives, traditionally pair-level training samples, as slot-level signals that encourage specialization.
• Real-world applications:
  • Natural-language search over large video archives, where a query must be matched to the correct clip among visually similar clips.
  • Video library and media-asset management, where editors locate footage by describing actions or objects.
  • Assistive or accessibility tools that retrieve video descriptions from free-text queries.
  • Image search and tagging pipelines, since the mechanism also transferred to text-image retrieval on COCO and Flickr30K.
• Industry relevance: MARS keeps a dual-encoder structure, so video representations can be precomputed and stored offline, while per-query online computation stays the lowest among the compared methods. That combination of accuracy and low online cost matters for production retrieval systems where cross-modal reranking is too expensive at query time.

Future Directions

• Adaptive slot interactions. The authors list exploring interactions among slots, rather than only uniform aggregation and independent matching, as future work.
• Temporal reasoning and moment-level retrieval. The paper explicitly focuses on video-level retrieval and does not address temporal grounding or moment retrieval.
• Broader and harder retrieval scenarios. The limitations section notes the benchmarks may not reflect open-domain settings with much longer videos, noisy descriptions, or diverse user queries.
• Other modalities. Applicability to audio-language or 3D-language retrieval remains unexplored, despite the demonstrated extension from video to images.

Target Audience

Researchers and engineers working on cross-modal retrieval, multimodal embedding models, or MLLM adaptation. It is most useful for readers already comfortable with contrastive retrieval training and transformer internals, and for practitioners who need a retrieval method that combines strong accuracy with a precomputable, dual-encoder inference pipeline. Readers primarily interested in temporal grounding, moment retrieval, or non-video modalities will find the scope deliberately narrow.

Authors’ abstract

Text-video retrieval requires representations that can distinguish videos with similar scenes, actions, and temporal patterns. Recent multimodal large language models have been adapted as embedding models, but they often represent each input using a single token from the final layer. This can compress diverse video-text cues into a single vector and limit fine-grained retrieval. To address this limitation, we propose MARS, a multi-layer and multi-slot embedding framework for text-video retrieval. MARS constructs multiple adaptive representation slots by combining hidden states from different decoder layers, compares corresponding text and video slots, and aggregates their similarities for retrieval. To better handle confusing candidates, we further introduce a hard-negative-aware slot specialization objective that encourages the slots to capture discriminative matching cues. Experiments on four text-video retrieval benchmarks show that MARS achieves state-of-the-art results in both direct similarity-based retrieval and reranking settings. Ablation studies and analyses demonstrate that multi-layer fusion, multiple slots, and hard-negative-aware slot specialization provide complementary gains. Code is available at https://github.com/sejong-rcv/MARS.