ShotFinder: Imagination-Driven Open-Domain Video Shot Retrieval via Web Search

AI summary

Overview

Research area: Multimodal information retrieval / computer vision — specifically open-domain video shot retrieval and temporal localization by multimodal large language models (MLLMs) acting as web-search agents.

Technical level: Intermediate. The concepts (shots, keyframes, temporal grounding, retrieval pipelines) are explained clearly in the paper, but familiarity with MLLMs, search-agent pipelines, and video grounding benchmarks helps.

Scope in one sentence: This paper introduces ShotFinder, a 1,210-sample YouTube benchmark for retrieving and temporally localizing video shots described in natural language under five single-factor constraints, plus an imagination-driven three-stage retrieval pipeline evaluated against it.

What This Paper Is About

Video editors who need a specific clip must currently scrub through footage manually, because no system reliably turns a written description into the right shot at the right timestamp. While language models have become good at text and static-image retrieval, video shot retrieval — which involves temporal structure, audio, visual style, and long redundant video context — has no systematic benchmark or analysis. The authors build that benchmark and a baseline method, then measure how far current models still are from human performance.

Key Contributions

1. The ShotFinder benchmark: the first benchmark for open-domain video shot retrieval, with 1,210 human-verified samples drawn from YouTube across 20 topic categories. It formalizes editing needs as keyframe-oriented shot descriptions with five controllable single-factor constraints: Temporal, Color, Visual Style, Audio, and Resolution (six task settings in total).

2. A constraint-aware data construction pipeline: LLM-generated descriptions (Gemini-3-Pro) refined by 10 computer-science researchers, combined with a topic-allocation strategy that routes each constraint type to topics where that constraint is naturally meaningful (e.g., Audio to TV series, documentaries, dance, animal footage; Resolution to fashion, movies, vlogs, fitness).

3. The ShotFinder method: a text-driven three-stage pipeline — (1) query expansion via "video imagination," (2) candidate video retrieval via a web search engine, and (3) description-guided shot localization using MLLMs with shot segmentation and adaptive keyframe sampling.

4. A keyframe-based MLLM voting evaluation protocol, validated against human judgment, after finding CLIP-style similarity metrics unreliable for fine-grained description-constraint matching.

Main Findings

• Humans vastly outperform all models. Human performance averages 88.5% accuracy (85.1% on Shot, 91.6% Temporal, 91.4% Color), while the best model reaches only 26.9%.

• GPT-5.2 leads the model field at 26.9% average, strongest in Temporal and Style, but weak in Color. Gemini-3-Pro follows at 24.4% (best in Temporal and Audio). GPT-5-mini (15.2%) and Claude-4.0-Sonnet (18.1%) trail substantially.

• Open-source models are competitive but not leading. Qwen3-VL-235B-A22B averages 20.6% and slightly beats Gemini-3-Pro on Color; Qwen3-Omni-30B-A3B averages 20.0% and performs well on Temporal and Audio. VideoLLaMA3-7B is lowest at 14.7%.

• Difficulty is uneven across constraints. Temporal localization is the most tractable task for nearly every model; Color and Style remain the hardest, with Style showing the largest gap between strong and weak models.

• Bigger models are not automatically better. Model scale does not produce consistent gains, implying architecture and multimodal alignment matter more than raw parameter count in this setting.

• Video imagination is the single most valuable component. Asking the model to first imagine the full video the shot likely belongs to — instead of keyword-extracting from the shot description directly — raises average accuracy by 10.7 points, with the largest gains on Resolution and Audio (+12.5 each). This bridges the granularity mismatch between a shot-level description and video-level metadata such as titles and tags.

• More queries and more candidates help, but unevenly. Increasing search queries from 1×2 to 3×2 improves Shot and Style markedly, with smaller Color gains. Increasing downloaded candidates from 2×1 to 2×3 improves overall performance, particularly Shot and Audio, but Resolution barely improves — suggesting it depends on inherent video quality rather than candidate quantity.

• Retriever choice matters modestly. Google Search, DuckDuckGo, and YouTube Search all land between 20.7% and 25.0% average; Google is best overall, DuckDuckGo better on Temporal and Color but worse on Shot and Audio.

• The pipeline is reproducible and evaluation is robust. Three full runs show low variance (average variance 1.14). Paraphrased evaluator prompts produce near-identical scores (average variance 0.11, five of six categories at zero). Keyframe-based scoring agrees with full-shot human judgment 98% of the time on 50 sampled instances.

• The pipeline is I/O-bound. Average per-sample latency is about six minutes, dominated by video downloading (40.4%), then search (29.3%) and temporal grounding (21.1%).

• Failures cluster into three stages. Generator errors (misread input → bad search queries), Retriever errors (good queries but no valid or relevant videos found), and Localizer errors (frame sampling mistakes or model judgments contradicting human annotations).

Methodology in Plain English

Building the benchmark. The authors crawled YouTube videos across 20 topic categories. Gemini-3-Pro generated candidate shot descriptions — one plain descriptive version plus versions with one added constraint each (temporal order relative to surrounding shots, color tone, visual style like live-action versus animation, audio content, and resolution). Ten researchers then manually checked and corrected every description for semantic, factual, and constraint accuracy, yielding 1,210 verified samples.

Deciding how to score. They first tried CLIP-style image-text similarity, but it disagreed badly with human judgment on fine-grained constraints. They switched to having three strong MLLMs (Gemini-2.5-Pro, GPT-5.2, Claude-4.0-Sonnet) vote on whether the retrieved keyframe matches the description and constraints, using the shot's keyframe as a stand-in for the full shot. A side experiment confirmed keyframes agree with full-shot human judgment 98% of the time.

The retrieval pipeline. Given a description, the Generator asks the model to imagine what complete video the shot probably came from, then derives search keywords from that imagined video — rather than summarizing the description directly. The Retriever sends those keywords to a search engine (Google by default), filters to accessible YouTube URLs, and downloads the videos. The Localizer uses TransNetV2 to cut each video into shots, extracts the midpoint frame of each shot, and asks an MLLM which frame matches the description (with audio provided for Audio tasks), outputting N/A if nothing matches.

The experiments. Each model under test generates two search queries with two URLs each, at temperature 0, and is scored by the three-model voting protocol. Beyond the main table, the authors ablate the number of queries, the number of downloaded candidates, whether video imagination is used, and which search engine is used.

Why This Matters

Research impact. ShotFinder fills a genuine gap: prior work either retrieves text/static images, or does video temporal grounding assuming the target video is already in hand. This paper couples open-domain retrieval with temporal grounding for the first time and shows that the two together are far harder than either alone — establishing a concrete, quantified capability gap for MLLM research to attack.

Real-world applications:

• Video editing and post-production: letting editors describe the footage they want in natural language instead of scrubbing timelines and memorizing timestamps, lowering the skill barrier for newcomers.
• Stock footage and media asset search: matching a script or storyboard line directly to licensable clips that satisfy narrative, color, style, and technical requirements.
• News and documentary research: locating archival b-roll of a described moment, action phase, or setting from open web video.
• Advertising and social content production: assembling footage sets with consistent color grading, visual style, or resolution without manual review.

Industry relevance. The pipeline is search-agent shaped — a language model reasoning about queries, calling an external search tool, and grounding results in media. The paper's finding that "video imagination" adds 10.7 points is a directly transferable design lesson for any agent bridging fine-grained user intent and coarse-grained web metadata. The latency breakdown (I/O-bound, dominated by downloading) also signals where engineering investment is needed to make such systems production-viable.

Future Directions

• Move beyond keyframes. The authors acknowledge that keyframes approximate shots; handling full video clips with motion, editing rhythm, and audio-visual interplay end-to-end is a natural next step.

• Multi-factor constraints. The benchmark deliberately isolates one constraint at a time. Real editing demands combined constraints (e.g., a warm-toned 1080p 3D-animated shot that follows a dialogue scene), which the authors flag as a harder, more realistic problem.

• Multi-turn and iterative retrieval. Query generation is currently single-turn and retrieval uses only basic usability filtering. Iterative query refinement and richer filtering using webpage HTML metadata could improve recall and precision.

• Closing the gaps that matter most. Color and Style are the weakest categories across all models; targeted work on fine-grained color perception and style recognition in retrieval is the clearest open problem. Broader model coverage as stronger MLLMs become available is also planned.

Target Audience

Researchers and engineers working on multimodal retrieval, video-language models, and web-search agents; benchmark designers interested in constraint-controlled task construction and LLM-assisted evaluation; practitioners building video editing, media asset management, or footage search tools; and anyone assessing how far multimodal models currently are from human-level understanding of video content in open-domain settings.

Authors’ abstract

In recent years, large language models (LLMs) have made rapid progress in information retrieval, yet existing research has mainly focused on text or static multimodal settings. Open-domain video shot retrieval, which involves richer temporal structure and more complex semantics, still lacks systematic benchmarks and analysis. To fill this gap, we introduce ShotFinder, a benchmark that formalizes editing requirements as keyframe-oriented shot descriptions and introduces five types of controllable single-factor constraints: Temporal order, Color, Visual style, Audio, and Resolution. We curate 1,210 high-quality samples from YouTube across 20 thematic categories, using large models for generation with human verification. Based on the benchmark, we propose ShotFinder, a text-driven three-stage retrieval and localization pipeline: (1) query expansion via video imagination, (2) candidate video retrieval with a search engine, and (3) description-guided shot localization. Experiments on multiple closed-source and open-source models reveal a significant gap to human performance, with clear imbalance across constraints: temporal localization is relatively tractable, while color and visual style remain major challenges. These results reveal that open-domain video shot retrieval is still a critical capability that multimodal large models have yet to overcome.