Fusionista2.0: Efficiency Retrieval System for Large-Scale Datasets

AI summary

Overview

Research area: Computer Vision and Multimodal Information Retrieval — specifically interactive video search systems built for competition-scale, multi-terabyte datasets.

Technical level: Intermediate. The paper assumes familiarity with CLIP-style embeddings, OCR/ASR pipelines, and Vision-Language Models, but the architectural choices are explained clearly enough for a reader with general machine learning background.

Scope: A system paper describing the re-engineering of an interactive video retrieval system for speed and usability, evaluated on the V3C dataset for the Video Browser Showdown 2026.

What This Paper Is About

The Video Browser Showdown (VBS) is a biennial competition where teams must retrieve specific moments from enormous video collections within strict time limits, using three query styles: Known-Item Search, Ad-hoc Video Search, and Visual Question Answering. As the target datasets grew into tens of thousands of videos (the V3C set alone has over 28,000 videos, plus specialized sets like Marine Video and LapGynLHE), the authors' prior systems became too slow — costly preprocessing, heavyweight models, and a cumbersome interface all became bottlenecks.

Fusionista2.0 is the authors' response: a rebuilt version of their Fusionista and Fustar systems in which every module (preprocessing, text search, OCR, ASR, question answering, reranking) was replaced with a faster, lighter alternative, and the entire user interface was redesigned for non-expert users.

Key Contributions

1. An all-in-one ffmpeg-based preprocessing pipeline. The previous multi-model keyframe selection stack (CLIP-B/32 embeddings, TransNetV2 scene detection, clustering) was replaced with deterministic extraction of intra-coded frames via ffmpeg, plus timestamp-to-frame mapping for reproducibility. This dramatically cuts GPU and memory costs at scale.

2. An ensemble text-to-image retrieval module. Two CLIP variants — SigLIP-400M and CLIP-ViT-5B — are combined with a fixed weighted sum, with the mixing parameter chosen empirically from a 50-participant user study.

3. A lightweight, scalable OCR/ASR and QA stack. Vintern-1B-v3.5 replaces PaddleOCR for multilingual scene text, faster-whisper replaces Whisper for transcription (roughly 4× faster), and InternVL-1B-ffn6-Seq is selected as the question-answering model after benchmarking five sub-1B-parameter VLMs.

4. A clarification-based reranking module and a redesigned UI/UX. GPT-4o generates three yes/no clarification questions from the user query; a vision-language model answers them per candidate image, and results are reordered by the number of affirmative answers. The interface was migrated from Create React App to Vite, made WCAG-compliant using shadcn/ui, and equipped with virtual scrolling, sidebar navigation, grouped results, and batch operations.

Main Findings

• Retrieval time dropped by up to 75% compared to the prior system, while accuracy and user satisfaction both improved — the central claim of the paper.
• Ensemble weighting matters measurably. In the 50-participant person-matching test, the α = 0.7 blend of SigLIP-400M and CLIP-ViT-5B let 43 participants find the target at top-1, versus 38 for SigLIP alone and 33 for ViT-5B alone. The weighting curve is non-monotonic, with a dip at α = 0.8 (29 people).
• Bigger is not better for time-critical QA. InternVL-1B-ffn6-Seq achieved the best balance across counting accuracy (0.84), image-answer accuracy (0.79), and video-answer accuracy (0.75) at an average of 4.80 s per query. LLaVA-0.5B-ffn6 was far slower (10.50 s) at lower accuracy, and SmolVLM-0.5B-ffn6 was fastest (1.07 s) but weak on image answers (0.52).
• Most VBS audio contains ambient sound, not speech. This observation justified dropping a large Whisper model in favor of a speed-optimized variant without meaningfully harming transcript quality.
• Models ≥7B parameters are impractical here. The authors explicitly position lightweight VLMs as handling routine queries (counting, attribute recognition, text extraction) while keeping humans in the loop for complex reasoning.

Methodology in Plain English

The authors treated the system as a pipeline and asked, at each stage, whether a lighter component could do the same job faster.

For preprocessing, instead of running several neural models over every video to pick "representative" keyframes, they simply asked the video codec for its own structural keyframes (the intra-coded frames) using the standard ffmpeg tool. This is deterministic, cheap, and good enough.

For searching by text description, they run two complementary CLIP models over every frame and average their similarity scores. Because one model may catch things the other misses, the blend outperforms either alone — but only if the blend is weighted correctly, which they determined by testing 50 real users rather than relying on a benchmark.

For reading on-screen text and transcribing speech, they swapped in much smaller models: Vintern-1B-v3.5 for OCR (which also reasons about partially occluded characters) and faster-whisper for ASR. For answering questions about a video, they built a 200-question benchmark from past VBS queries across three categories, then measured five compact VLMs on it to pick the best speed-accuracy trade-off.

For reranking, they used GPT-4o to turn the user's free-text query into three concrete yes/no questions (for example, "Is there a dog in the scene?"), then had a vision-language model check each candidate image against those questions. Images answering more questions affirmatively move up the list.

Finally, the interface was rebuilt on faster tooling, made accessible, and reorganized so users see fewer redundant frames from the same video.

Why This Matters

Impact on research: This is a systems-efficiency paper rather than a new-model paper, and its value lies in demonstrating that a carefully chosen ensemble of small models, combined with a well-designed interface, can beat a heavier pipeline on an interactive benchmark. It provides concrete evidence that scale-up is not the only path to better retrieval performance under real constraints.

Real-world applications:

• Broadcast and newsroom archives — journalists locating a specific clip of a politician, event, or quote across decades of footage within seconds.
• Media production and stock footage — editors searching by described action, on-screen text, or spoken dialogue rather than manually tagged metadata.
• Medical and scientific video review — the paper's own datasets include laparoscopic gynecology footage (LapGynLHE) and marine video, where domain experts need to find specific procedural or biological moments.
• Surveillance and forensic review — locating individuals, vehicles, or specific textual signage in long camera recordings, where response time is operationally critical.

Industry relevance: Any organization sitting on large video libraries — streaming platforms, cloud storage providers, e-learning companies, sports leagues — faces the same cost trade-off between retrieval quality and compute. Fusionista2.0's component substitutions (lighter OCR, faster ASR, sub-1B VLMs) are directly transferable cost-reduction strategies, and its observation that lightweight models suffice for routine queries informs how human-in-the-loop review tools should be designed.

Future Directions

• Automating the reranking question generation. The pipeline currently depends on GPT-4o to produce clarification questions, an external API call that introduces latency and cost; a distilled local model could remove that dependency.
• Adaptive blending rather than a fixed α. The optimal CLIP weighting may vary by query type or dataset, and the current fixed α = 0.7 (chosen from a single person-matching study) may not generalize across the Marine Video and LapGynLHE domains.
• Extending multilingual coverage. Vintern-1B-v3.5 was selected partly for low-resource-language performance, but the evaluation focuses on a narrow benchmark; broader language coverage and cross-lingual query handling remain open.
• Systematic human-in-the-loop evaluation. The current evidence rests on a 50-person study and competition results; larger controlled studies would clarify when lightweight models should defer to human judgment and how that handoff should be triggered.

Target Audience

This paper suits researchers and engineers building interactive multimedia retrieval systems, particularly those working under latency or compute budgets. It is also valuable for competition participants in VBS or similar benchmarks (TRECVID, Video Search Showdown-adjacent challenges) who need a practical reference for which off-the-shelf components to assemble. Product teams at media, streaming, or archival organizations evaluating retrieval prototypes will find the module-by-module trade-off justifications directly actionable. Readers seeking novel model architectures or theoretical advances will find little here — the contribution is engineering and interface design.

Authors’ abstract

The Video Browser Showdown (VBS) challenges systems to deliver accurate results under strict time constraints. To meet this demand, we present Fusionista2.0, a streamlined video retrieval system optimized for speed and usability. All core modules were re-engineered for efficiency: preprocessing now relies on ffmpeg for fast keyframe extraction, optical character recognition uses Vintern-1B-v3.5 for robust multilingual text recognition, and automatic speech recognition employs faster-whisper for real-time transcription. For question answering, lightweight vision-language models provide quick responses without the heavy cost of large models. Beyond these technical upgrades, Fusionista2.0 introduces a redesigned user interface with improved responsiveness, accessibility, and workflow efficiency, enabling even non-expert users to retrieve relevant content rapidly. Evaluations demonstrate that retrieval time was reduced by up to 75% while accuracy and user satisfaction both increased, confirming Fusionista2.0 as a competitive and user-friendly system for large-scale video search.