MUVR: A Multi-Modal Untrimmed Video Retrieval Benchmark with Multi-Level Visual Correspondence

AI summary

Overview

Research area: Computer vision / multimodal video retrieval, specifically benchmarking retrieval of untrimmed long videos using multimodal queries.

Technical level: Advanced

Scope: The paper introduces MUVR, a benchmark and task definition ("Multi-modal Untrimmed Video Retrieval") built from 53,462 Bilibili videos, 1,050 video-centric multimodal queries, and 84,035 matches, spanning five content partitions and six levels of "visual correspondence," together with three evaluation versions (Base, Filter, QA) and a new Reranking Score for multimodal large language models (MLLMs).

What This Paper Is About

Searching a large library of real, unedited long videos is hard because a single text query often cannot describe the visual details a user actually wants, and because most existing benchmarks assume short, pre-trimmed clips and one-to-one matching. The authors build a benchmark where the query itself is a video (supplemented by long text descriptions, optional tag prompts, and mask prompts) and where one query can legitimately match many videos. Their goal is to define a realistic retrieval task, provide matching criteria that cover many video categories, and measure how badly current retrieval models and MLLMs perform on it.

Key Contributions

1. A new task and benchmark (MUVR). The paper proposes Multi-modal Untrimmed Video Retrieval with a video-centric multimodal query format (video query + text description + tag prompt + mask prompt), one-to-many retrieval, and untrimmed target videos, organized into five partitions (news, region, instance, dance, others) covering diverse video categories.
2. Multi-level visual correspondence as a matching criterion. Six levels are defined — copy, event, scene, instance, action, and others — describing how a segment of a query video relates to a segment of a target video, replacing the narrower criteria used by prior tasks (global semantic match, video modification, almost-identical segment, same-incident segment).
3. Three benchmark versions plus a new metric. MUVR-Base, MUVR-Filter (which adds tag-prompt-based hierarchical filtering), and MUVR-QA (200 query–target relevance questions), along with a Reranking Score that rewards keeping true positives and removing false positives.
4. Extensive evaluation and error analysis. Three state-of-the-art video retrieval models, six image-based VLMs, and ten MLLMs are evaluated, with findings about which query type, partition, and model family performs best.

Main Findings

• Retrieval performance is low overall. On MUVR-Base with multimodal queries, the strongest model, EVA-CLIP, reaches 58.0 mAP, 44.6 uAP, 73.0 R@200, 82.5 R@500, and 92.3 R@2000. On MUVR-Filter it reaches 34.0 mAP with the tag prompt and 32.9 mAP without it.
• Model scale and training data drive average performance. CLIP-based models improve from ResNet50 to ViT-H-14 backbones, and EVA-CLIP leads on nearly all metrics, which the authors attribute to larger parameter count and training data volume.
• Video retrieval frameworks can beat their backbones. S2VS, built on CLIP (RN50x4) features, improves from 34.2% to 47.2% mAP and from 16.1% to 36.6% uAP for video queries, showing that pre-extracted VLM frame features can be improved through retrieval-framework training.
• Video queries matter more than text, and combining them helps most. Pure video queries generally outperform pure text queries, while multimodal queries give the largest gains — except on the Instance partition, where small key objects mean video queries introduce more irrelevant content and underperform text.
• Different partitions reward different capabilities. EVA-CLIP performs best on News, Instance, and Region (static spatial understanding), while InternVideo2 and S2VS perform best on Others and Dance (dynamic temporal understanding).
• CoVR is poorly suited to this setting. Because it supports only a 1-frame query video input, it performs worse on untrimmed videos.
• Tag prompts help strong models but not weak ones. On MUVR-Filter, ignoring the tag prompt slightly hurts recall but significantly reduces mAP; BLIP shows only marginal gains with tag prompts, indicating small models struggle to comprehend them, and all models improve only modestly on the Instance and Dance partitions.
• MLLMs discriminate, but do not reliably rerank. Some MLLMs exceed 70% accuracy on questions without tag prompts, but performance drops substantially when tag prompts are included. Best reported results include GPT-4o at 65.0% accuracy and 0.19 Reranking Score (6 frames, All) and InternVL2 at 58.5% accuracy and -0.23 Reranking Score (6 frames, All, multi-image comparison).
• More frames help, up to a point. InternVL2.5 improves from 52.0% to 57.0% accuracy when going from 1 to 6 frames in multi-image comparison, but inference time rises from 1.49s to 4.23s, and gains diminish at 12 frames.
• Multi-image and mask prompt abilities improve reranking. Models that jointly process query and target frames (InternVL, MiniCPM) consistently outperform text-only comparison on Reranking Score, and VideoRefer, which can use the mask prompt, performs better with it (0.05 to 0.07 Reranking Score on All).
• The benchmark is robust to annotation noise. Randomly increasing or decreasing positive samples by 5%, repeated five times, changes scores only slightly (e.g., EVA-CLIP from 63.4±0.1 to 56.0±0.4 on News) while the relative ranking of methods stays stable.
• MUVR-Base is larger and broader than prior fine-grained retrieval benchmarks. It has 350 topics, 1,050 queries, 84,035 matches, 53,462 videos, and 1,762 hours from Bilibili (2024), compared with, for example, FIVR's 100 topics, 100 queries, 12,300 matches, 225,960 videos, and 7,100 hours from YouTube (2017).

Methodology in Plain English

The team started from 350 search topics derived from Bilibili trending keywords and split them across five partitions. For each topic they collected the top 100 search results, dropped videos longer than 6 minutes, cropped long videos into 2-minute untrimmed clips, resized them to 336 pixels on the long edge, and downsampled to 6 fps.

Professional annotators then picked three representative, visually distinct videos per topic as video queries and wrote detailed text descriptions of the key visual content and retrieval needs. Matches were annotated only within the same topic, cross-topic candidates were screened for similarity using BLIP2 features (the query video had to be unrelated to the 10 most similar videos from other topics), and every video was annotated twice with conflicting labels excluded. This produced MUVR-Base, averaging about 80 verified positives per query.

On top of that, annotators designed 3 to 10 tags per topic capturing shared attributes of positives, such as animation versus live-action style, first-person perspective, or indoor versus outdoor settings. Tag prompts then enable hierarchical filtering, giving MUVR-Filter; the paper also reports an upper bound using AND/OR combinations of tag prompts. Mask prompts produced with SAM2 mark fine-grained regions in query videos. For MUVR-QA, the authors selected 200 hard cases where EVA-CLIP scored below 0.05 mAP, pairing the highest-scoring true match with the highest-scoring false match.

Evaluation uses mAP, uAP, and Recall@k on MUVR-Base; mAP and Recall@k on MUVR-Filter; and Accuracy plus the new Reranking Score on MUVR-QA. The Reranking Score assigns +1 when a model keeps the true positive and removes the false positive, 0 when it keeps both (no reranking action), -1 when it

Authors’ abstract

We propose the Multi-modal Untrimmed Video Retrieval task, along with a new benchmark (MUVR) to advance video retrieval for long-video platforms. MUVR aims to retrieve untrimmed videos containing relevant segments using multi-modal queries. It has the following features: 1) Practical retrieval paradigm: MUVR supports video-centric multi-modal queries, expressing fine-grained retrieval needs through long text descriptions, video tag prompts, and mask prompts. It adopts a one-to-many retrieval paradigm and focuses on untrimmed videos, tailored for long-video platform applications. 2) Multi-level visual correspondence: To cover common video categories (e.g., news, travel, dance) and precisely define retrieval matching criteria, we construct multi-level visual correspondence based on core video content (e.g., news events, travel locations, dance moves) which users are interested in and want to retrieve. It covers six levels: copy, event, scene, instance, action, and others. 3) Comprehensive evaluation criteria: We develop 3 versions of MUVR (i.e., Base, Filter, QA). MUVR-Base/Filter evaluates retrieval models, while MUVR-QA assesses MLLMs in a question-answering format. We also propose a Reranking Score to evaluate the reranking ability of MLLMs. MUVR consists of 53K untrimmed videos from the video platform Bilibili, with 1,050 multi-modal queries and 84K matches. Extensive evaluations of 3 state-of-the-art video retrieval models, 6 image-based VLMs, and 10 MLLMs are conducted. MUVR reveals the limitations of retrieval methods in processing untrimmed videos and multi-modal queries, as well as MLLMs in multi-video understanding and reranking. Our code and benchmark is available at https://github.com/debby-0527/MUVR.