Adapting MLLMs for Nuanced Video Retrieval

AI summary

Overview

• Research area: Computer vision / multimodal retrieval — adapting Multimodal Large Language Models (MLLMs) into video-text embedding models.
• Technical level: Intermediate (assumes familiarity with contrastive embedding models, retrieval benchmarks, and MLLM architectures).
• Scope: The paper introduces TARA (Text Adapted Retrieval Alignment), a text-only fine-tuning recipe that makes MLLMs better at nuanced video retrieval across temporal, negation, and multimodal aspects.

What This Paper Is About

Text-to-video search engines need to rank videos by relevance to a query, but real queries are nuanced: they depend on the order of events ("opening a door" versus "closing a door"), on what the user does not want ("not on grass"), or combine a video example with a text edit instruction. The authors' goal is a single embedding model that handles all three kinds of nuance, built by repurposing an MLLM that was originally trained only to generate text. Their central claim is that this can be done by fine-tuning on text triplets alone, with hard negatives chosen to inject each nuance, without ever training on paired video data.

Key Contributions

1. TARA (Text Adapted Retrieval Alignment): a method for curating text triplets with carefully selected hard negatives to fine-tune MLLMs into strong retrieval encoders.
2. Consistent gains across base models: applying TARA to any base MLLM always improves its nuanced retrieval performance, often substantially, and reaches state-of-the-art results on CiA, RTime, NegBench, and CoVR (WebVid-CoVR).
3. No harm on standard benchmarks: text-only fine-tuning also improves the video tasks of standard benchmarks such as MMEB-V2, and a simple concatenation of TARA with Qwen3-VL-Embedding outperforms the latter on average.
4. An explanation via the modality gap: the paper shows text-only fine-tuning reduces the modality gap between text and video embeddings, which re-organizes the joint embedding space and explains the retrieval gains, supported by both empirical measurements and a theoretical bound.

Main Findings

• Text-only training is enough, and is fast: the curated dataset, called NLI-Nuance, contains 20,000 triplets (8K from NLI plus 12K covering temporal, negation, and multimodal nuances). Fine-tuning a 7B model on 8 RTX A6000 GPUs takes less than an hour. Only the LLM weights are fine-tuned; vision and projection networks are frozen. Training runs 2 epochs with batch size 768 and base learning rate 2e-5, using F=16 uniformly spaced frames at inference.
• Temporal nuance (CiA): Tarsier 2 + TARA reaches 88.9 mAP on the SSv2 chiral split, 66.7 static, and 58.6 all; on EPIC it reaches 81.1 chiral, 45.6 static, 38.9 all; on Charades 71.4 chiral, 38.6 static, 29.0 all. The paper notes TARA always improves over the base model for chiral action retrieval, with smaller gains when the base model was already trained contrastively on multimodal data (Qwen3VL-Emb.) and larger gains for autoregressively trained bases.
• Temporal nuance (RTime): Tarsier 2 + TARA achieves 67.2 R@1 for T2V and 77.9 for V2T, beating methods fine-tuned on the RTime training set, and improving on its base model's 58.8 / 59.5.
• Negation nuance (NegBench): Tarsier 2 + TARA reaches 76.7 R@5 and 73.6 R-Neg@5 on COCO, and 65.1 / 65.0 on MSR-VTT, versus 33.3 / 21.5 and 25.6 / 18.9 for the untuned base model, and better than NegCLIP and CLIP variants fine-tuned on Conceptual Captions.
• Multimodal nuance (WebVid-CoVR): evaluated zero-shot on the 2,556 query-video WebVid-CoVR test set, Tarsier 2 + TARA scores 66.3 R@1, 86.7 R@5, and 91.5 R@10, outperforming methods fine-tuned on WebVid-CoVR (the best of which reaches 59.8 / 83.8 / 91.3).
• Standard benchmarks (MMEB-V2): across 10 video classification and retrieval datasets, Tarsier 2 + TARA improves its base model from 27.4 to 66.1 average classification and from 16.4 to 45.9 average retrieval, second overall only to Qwen3-VL-Embedding (77.5 and 58.9). The concatenated TARA ⊕ Q3VLE reaches 78.5 and 59.5, exceeding Qwen3-VL-Embedding.
• Controlled comparison: with the same Qwen2VL-7B base, TARA averages 56.8 across the nuanced benchmarks versus 51.9 for the second-best method (CaRe), despite using text only. The paper describes this as a large margin over the second-best method in its own framing of the table.
• Modality gap exists even in shared-backbone MLLMs: measured on 1,000 MSRVTT video-text pairs using the definition from prior work, Qwen2VL shows a clear separation between modalities, which the authors attribute to video and text tokens arriving through different pathways (a pretrained vision encoder plus MLP projection versus learned text embeddings).
• The EOL prompt alone does not close the gap for video: contrary to earlier image-text findings with LLaVA-Next-8B, using an "Explicit One-word Limitation" prompt with Qwen2VL does not dissolve the modality gap for video-text data; text-only fine-tuning does. Methods trained on large amounts of video-text data (CaRe, Qwen3-VL-Embedding) also show a reduced gap.
• Theory supports the recipe: under the Modality Gap Assumption, the paper shows ℓ_text-only ≤ ℓ^t2v ≤ e^{ν²} · ℓ_text-only pointwise per triplet, and hence L_text-only ≤ L_t2v, meaning driving the text-only loss to zero drives the text-to-video loss to zero at the same rate up to a noise-dependent constant.
• Mechanism: the authors attribute the effect to uniformity pressure — unimodal contrastive learning spreads embeddings over a larger subspace of the hypersphere, moving both text and video centroids toward the origin. TARA reduces the uniformity loss in both modalities compared to the base model, and the reduction in gap tracks both the training loss and improved downstream retrieval on SSv2.
• Embeddings become semantically interpretable: feeding TARA embeddings through the LM head yields top tokens matching the relevant action (for example "closing the box"), whereas the base model produces a semantically meaningless first token. For chiral pairs, the base model shows δ ∼ 0 (poor text-video association) while TARA achieves much higher δ.

Methodology in Plain English

The authors start from an MLLM that generates text, and turn it into an embedding model by prompting it to "summarize the video in one word" (or the sentence, or the edited video) and using the final-layer hidden state of the next token as the embedding. Query-video similarity is then just the dot product of these embeddings.

The key move is how they train it. Instead of the usual two-stage recipe (contrastive text training followed by large-scale multimodal training), they fine-tune with a contrastive loss on text triplets only — an anchor sentence, a positive match, and a hard negative — and they curate the negatives so each one targets a specific nuance:

• Temporal: verb-object pairs are extracted from Ego4D, filtered to those with a "chiral" verb (a verb with a temporal opposite) using a pre-compiled list generated by Claude. Positives share the chiral verb with the anchor; hard negatives are generated by prompting Qwen 3 to produce temporally opposite sentences (the appendix describes using Qwen3-1.7B with in-context samples to generate antonyms for 425K sentences, with LLM-as-a-Judge verification followed by manual review).
• Negation: triplets from the NLI dataset are filtered to keep those whose hard negative contains a negator such

Authors’ abstract

Our objective is to build an embedding model that captures the nuanced relationship between a search query and candidate videos. We cover three aspects of nuanced retrieval: (i) temporal, (ii) negation, and (iii) multimodal. For temporal nuance, we consider chiral actions that need distinguishing between temporally opposite actions like "opening a door" vs. "closing a door". For negation, we consider queries with negators such as "not", "none" that allow a user to specify what they do not want. For multimodal nuance, we consider the task of composed retrieval where the query comprises a video along with a text edit instruction. The goal is to develop a unified embedding model that handles such nuances effectively. To that end, we repurpose a Multimodal Large Language Model (MLLM) trained to generate text into an embedding model. We fine-tune it with a contrastive loss on text alone with carefully sampled hard negatives that instill the desired nuances in the learned embedding space. Despite the text-only training, our method achieves state of the art performance on all benchmarks for nuanced video retrieval. We also analyze how this improvement is achieved, and show that text-only training reduces the modality gap between text and video embeddings leading to better organization of the embedding space.