MultiVENT-Raw: A Benchmark for Retrieval and Reasoning over Raw Videos

AI summary

Overview

Research area: Multimodal video retrieval and retrieval-augmented generation (computer vision and natural language processing), focused on event-centric information access over unedited "raw" footage.

Technical level: Advanced — the paper assumes familiarity with dense retrieval, bi-encoders, late-interaction retrieval, reranking, vision-language models, and RAG evaluation metrics.

Scope: The paper releases MultiVENT-Raw, a multilingual benchmark of nearly 120,000 videos (over 5,300 hours) with 130 events and 222 queries for evaluating video retrieval and multi-video report generation, plus baseline results showing both tasks remain difficult.

What This Paper Is About

Online information is increasingly delivered as video, and much of it is raw footage — continuous, unscripted, unedited material from cell phones, hand-held cameras, or CCTV, uploaded directly to platforms. Unlike edited news footage, raw video lacks narration, captions, chyrons, graphics, and metadata, so the context needed to match a high-level user query to a clip must be inferred rather than read off the screen. Existing video retrieval benchmarks focus overwhelmingly on short, English, edited videos with simple queries, so the authors build a large multilingual benchmark of raw video with realistic, event-centric queries and both retrieval and generation tasks.

Key Contributions

1. MultiVENT-Raw, a large multilingual benchmark for retrieval and reasoning over raw video, spanning 8 languages (Arabic, Chinese, Farsi, Korean, Pashto, Russian, Spanish, Urdu), 130 events, 222 queries, and nearly 120,000 videos totaling over 5,300 hours.
2. MicroVENT, a smaller dev set of 23 events, 31 queries, and 933 total videos (272 annotated) for rapid system iteration, allowing MultiVENT-Raw to serve as a test set.
3. Human annotation resources: manual relevance judgments for all queries and roughly 5,800 manually written key facts ("claims") across 703 relevant videos, averaging 8.25 claims per video, with per-claim confidence scores and modality support labels.
4. Baseline results on retrieval and report generation, showing both tasks to be challenging even for recent multimodal models, with especially low absolute scores on citation metrics.

Main Findings

• Raw video retrieval is hard for general-purpose models. On the Full split, ColQwen Omni (3B) reaches nDCG@10 of 23.3 and R@100 of 39.2, while Qwen3-VL (8B) reaches nDCG@10 of 26.7 and R@100 of 45.6. OmniEmbed (7B) without domain fine-tuning performs worst at nDCG@10 of 18.0 and R@100 of 34.9.
• Domain-specific fine-tuning matters most. OmniEmbed-mv (7B), fine-tuned on the MultiVENT 2.0 train split, is the strongest first-stage retriever by a wide margin: nDCG@10 of 47.2, nDCG@20 of 48.9, R@20 of 51.6, and R@100 of 71.8 on Full; 49.7, 50.8, 51.9, and 67.2 on Core.
• Reranking helps, but not uniformly. Reranking OmniEmbed-mv with RankVideo (8B) gives 48.5 nDCG@10 and 57.2 R@20 on Full, and Gemma 4 (12B) zero-shot reranking gives 51.5 nDCG@10, 54.4 nDCG@20, and 57.3 R@20 on Full. On Core, reranking improves R@20 but does not have a consistent effect on nDCG.
• The Full split is harder than Core. The same models score lower on Full across the board, which the authors attribute to the semantically rich edited hard negatives present in Full but removed from Core.
• Report generation scores are low in absolute terms. Under MiRAGE with Qwen3.5-27B as judge and top-10 retrieved videos, TRACE lite consistently achieves the highest Information F1 (24.6 on Full and 25.2 on Core with OmniEmbed-mv; 26.2 on Full and 25.9 on Core with RankVideo reranking), while MARQUIS bullet tends to obtain the best Citation F1.
• Retrieval quality drives generation quality. The authors report roughly 6–8 point gaps in Information F1 between the best retrieval system (OmniEmbed-mv with reranking) and the worst (OmniEmbed), validating the importance of the retrieval stage.
• Citation is the weakest part of the pipeline. Citation precision, recall, and F1 values are very low across all systems and splits, confirming the difficulty of grounded report generation.
• Raw/edited classification is reliable. An ensemble of two zero-shot prompted VLMs, two fine-tuned generative VLM classifiers, and an FFmpeg-feature classifier achieved 91.4% Macro-F1 (Accuracy 95.1, Raw-F1 85.7) on the dev set, and was used to define the Core split by removing irrelevant videos labeled edited.
• Anomaly events add difficulty. The anomaly subset was curated from 322 public livestream cameras sampled at regular intervals over 11 months (June 2025 through April 2026), yielding 24,459 clips from cameras in over 20 countries; cameras with more than 4% anomalous clips were dropped. These events are even less likely to appear in model training data, and the relevant camera activity is embedded in tens or hundreds of minutes of similar footage.

Methodology in Plain English

The authors first curated topics. For "primary events," they searched news sites and social media (YouTube, TikTok, X) for recent, locally covered events in each of eight target languages, using both manual discovery and an automated Claude skill, and kept an event only if multiple raw videos depicting it could be found. For "anomaly events," they reviewed footage from a pool of livestream and CCTV cameras to establish each camera's normal activity profile, then flagged clips that deviated from it.

All videos were manually verified as raw and labeled relevant to their topic. To create realistic distractors ("hard negatives"), the team retrieved candidate videos from a larger collection using dense embeddings of visual keyframes, speech transcripts, and OCR text, ranked them by similarity to each query, and kept high-scoring clips that were not judged relevant. A second round of manual judgments was collected on the top 20 relevant videos per query as ranked by trained retrieval baselines. The team then trained a raw-versus-edited classifier on the MultiVENT 2.0 test set and used it to produce two collection splits: Full (all 118,802 original videos) and Core (59,196, with classifier-labeled edited videos removed).

For each topic, annotators wrote a persona (a job title and description for a hypothetical stakeholder) and queries. Each primary event gets two queries: a broad query written from the persona plus claims from all relevant videos, and a narrow query written from the persona plus relevant video titles only. Anomaly events get broad queries only. In total there are 222 queries. Linguists then watched each relevant video and wrote claims in English, with a 0–100 confidence score and tags for which modalities supported each claim (visual frames, on-screen text, speech, or non-speech audio), using the FOVEA annotation tool. Claims were post-processed with Claude skills to make them unique, atomic, and self-contained; to filter them into event, persona, and query subsets; and to add "negative claims" asserting that the collection provides no answer to a query facet.

For experiments, all videos were chunked into segments of at most 5 minutes (143,288 segments for Full, 75,147 for Core), and retrieval was indexed at the segment level. Retrieval is scored with nDCG@10, nDCG@20, R@20, and R@100. Report generation is evaluated with MiRAGE in both a RAG setting (using retrieved lists) and an Oracle setting (using the ideal ranked list), measuring information precision, recall, and F1 plus citation precision, recall, and F1.

Why This Matters

Raw video is now a dominant format for online information, but benchmarks and models have been built around edited, captioned, English-language clips. This benchmark shifts evaluation to the harder, more realistic case where context is implicit and must be inferred, and where the candidate pool is deliberately full of near-miss edited material. It also extends language coverage beyond prior MultiVENT work and introduces anomaly detection over continuous livestream footage.

Real-world applications:

• Crisis response and humanitarian monitoring, where footage from cell phones and CCTV documents disasters, protests, and incidents that may never receive edited news coverage.
• Journalistic verification and event reconstruction, where a reporter must find and corroborate footage of an event across many sources.
• Open-source intelligence and analyst workflows, where relevant activity is buried in hours of continuous camera footage and must be flagged or summarized.
• Platform content moderation and archive search, where systems must retrieve and summarize relevant raw clips from very large, multilingual video collections.

Industry relevance: The results are directly relevant to teams building video search, video RAG, and multimodal assistants. The finding that domain-specific fine-tuning (OmniEmbed-mv) yields a large gain over general-purpose retrievers, and that retrieval quality drives roughly 6–8 point swings in downstream report Information F1, gives concrete guidance on where to invest. The near-zero citation scores also indicate that grounded, attributable generation over video remains an unsolved production problem.

Future Directions

• Improving first-stage retrieval for raw video, particularly when edited near-duplicates are present in the collection, since Full consistently underperforms Core.
• Better multi-video RAG architectures that can handle very long model outputs, given that TRACE produced reports often of 10,000 sentences or more, causing context rot and instruction-following failures in claim verification.
• Raising citation precision and recall, which remain the weakest reported metrics across all systems and splits.
• Extending evaluation to the anomaly setting, where events must be detected within tens or hundreds of minutes of otherwise normal footage and are unlikely to appear in model training data.
• The paper does not report results on additional languages, other model families, or alternative evaluation judges beyond those described, leaving those comparisons open.

Target Audience

Researchers and engineers working on text-to-video retrieval, multimodal retrieval-augmented generation, and video understanding; benchmark developers interested in multilingual and raw-video evaluation; and practitioners building search or summarization systems over user-generated or surveillance-style video, who will benefit from the baseline numbers showing where current models fall short.

Authors’ abstract

Online information is increasingly consumed in video format. Much of this comes in the form of *raw video*: continuous footage taken on a cell phone, with a hand-held camera, or via CCTV, which is then directly uploaded to social media platforms and content sharing services. Whereas professional or even amateur-edited footage tends to feature scripted speech, chyrons, graphics, and metadata that help contextualize its subject matter, raw video typically contains none of these things, making it a much more challenging medium for information retrieval and machine understanding. To facilitate progress in this domain, we release MultiVENT-Raw, a multilingual collection of nearly 120,000 primarily raw videos (over 5,300 total hours), paired with 130 events and 222 event-centric queries, along with human-annotated video relevance judgments and human-extracted key facts for relevant videos. MultiVENT-Raw supports both a retrieval task---to identify videos in the collection relevant to a query event---and a generation task---to summarize event-related videos into a coherent report for a target user. We benchmark strong baselines on MultiVENT-Raw, showing both tasks to be challenging even for some of the latest multimodal models.