Caption-once, Frames-on-Demand: Visual-Need Routing for Budget-Aware Agentic Long Video Understanding

AI summary

Overview

Research area: Efficient long-video multimodal question answering (video QA), agentic MLLM pipelines, and edge-cloud inference for resource-constrained deployment.

Technical level: Advanced. The paper assumes familiarity with multimodal large language models (MLLMs), retrieval-augmented agent loops, frame sampling/visual token compression, and long-video benchmarks such as Video-MME and InfiniBench.

Scope: The paper proposes and evaluates CFD (Caption-once, Frames-on-Demand), an edge-cloud agentic framework that indexes a video into language memory once and retrieves raw frames only when a question specifically requires pixel-level evidence.

What This Paper Is About

Long videos (hours of footage) must be understood by AI systems running under tight compute, bandwidth, and latency budgets. Existing methods either compress visual tokens (risking loss of coverage and detail) or translate video into text memories (risking loss of fine-grained visual attributes such as color, identity, and on-screen text). This paper argues that text and pixels are good at different things — text preserves long-range temporal structure, pixels resolve perceptual attributes — and builds a system that allocates visual access selectively on a per-question basis instead of uniformly.

Key Contributions

1. Visual-Need Router. A per-query gating module that classifies a question as perceptual (appearance, on-screen text, spatial layout, attribute disambiguation) or temporal-structural (ordering, scene transitions, narrative), triggering raw-frame retrieval only for the former. This converts frame retrieval from an emergent pipeline byproduct into an explicit, query-conditioned cost.

2. Caption-once, Frames-on-Demand framework. A dual-track narrative index built in a single offline pass — an event-level story skeleton plus a clip-level micro-log — combined with the router and a fixed-capacity FIFO visual working memory, making per-query visual cost a tunable hyperparameter rather than an implicit function of interaction depth.

3. No online re-captioning. Unlike prior agentic video systems that invoke a captioner repeatedly during question answering, CFD builds all text memory offline and enriches events by composing already-cached clip captions through pure text operations, at zero additional visual cost.

4. Empirical validation. On Video-MME and InfiniBench, CFD reaches accuracy competitive with or better than prior agent-based methods while using roughly an order of magnitude fewer frames per question.

Main Findings

• Video-MME accuracy at very low frame cost: CFD scores 67.5 overall using 5.8 frames per question on average, exceeding agent-based baselines VideoLucy (64.7) and MemVid (64.0). Dense baselines use hundreds to 768 frames; Qwen3-VL-32B reaches 75.9 at 768 frames, a ceiling the authors explicitly do not claim to match.

• Temporal reasoning beats dense inference: On InfiniBench, CFD achieves the best Chronological Understanding score (55.10), surpassing 768-frame dense baselines such as Qwen3-VL (48.44) and Qwen2-VL (48.41), at only 14.8 frames per question on average.

• Attribute perception remains the weak spot: Global Appearance (58.90) and Character Actions (56.20) trail dense Qwen3-VL (70.54 and 67.04), confirming the visual-textual duality — a bounded on-demand frame budget does not fully substitute for exhaustive pixel sampling on attribute-binding tasks.

• Router reduces cost and improves accuracy: Within an already sparse pipeline, the Visual-Need Router cuts online frames from 6.63 to 5.76 per query (13.1% reduction) while raising accuracy from 67.2 to 67.5. Overall, the pipeline reduces frame usage from 768 to 6.63, a 99.1% reduction or roughly 116x.

• Selective routing avoids context dilution: Disabling the router (always retrieving frames) improves appearance-centric skills but degrades Chronological Understanding (51.70 vs. 55.10) and long-video accuracy (62.6 vs. 63.6), showing indiscriminate frame injection harms temporally structured questions.

• Component complementarity: Event memory alone is too sparse (57.3 overall, Global Appearance 9.30); dense clip memory alone reaches 65.5 overall but only 38.00 on Global Appearance. Text handles temporal structure; frames are needed for attributes.

• Larger captioners are not always better: Scaling the offline captioner from 2B to 8B improves overall accuracy (62.6 to 67.5), but scaling to 32B degrades long-video accuracy (63.6 to 61.0) due to verbosity accumulation across many events.

• Budget insensitivity: Sweeping the per-event frame cap and FIFO capacity over an 8x range changes overall accuracy by only 0.4%, supporting the claim that a few targeted frames suffice.

• Router trigger statistics: The router retrieves frames for 46.2% of questions and captures 127 of 133 (95.5%) cases where frames empirically improve the answer, per an outcome-based diagnostic oracle.

• Long videos benefit from more iterations: Long-video accuracy improves from 60.4 at T=1 to 64.0 at T=20; short-video accuracy saturates by T=2. The default is T=5.

• Clip-caption activation saves text context: CFD activates 66.3% fewer clip captions on average than an all-clip baseline while performing better overall.

Methodology in Plain English

The system splits work between an offline stage and an online stage.

Offline (edge): A lightweight vision-language model watches the video once and writes down two sets of notes. The first is an "event memory" — the video is cut at shot boundaries and merged into variable-length events of at least 60 seconds, each summarized as a structured story entry covering scene, entities, chronological flow, and retrieval tags. The second is a "clip memory" — independent fixed 30-second clips, each captioned at high granularity with one action per sentence. Both are cached and reused for every future question about that video; the video is never re-captioned.

Online (cloud): A stronger model answers questions through a loop. It first tries to answer using only the full event memory (the "story-first" attempt) — if confident, it returns immediately with one LLM call. Otherwise a localization agent picks the most relevant unexplored event, and all clip captions overlapping that event are injected into its text as nested details. The model re-answers. If still unsure, the Visual-Need Router decides whether looking at actual pixels would help: perceptual questions get frames, temporal questions do not. When frames are fetched, they are sampled at a low rate (about one frame per 10 seconds of the event), capped at 8 frames per event, and stored in a 16-frame FIFO queue. The model then re-answers with text and images together. The loop repeats up to 5 times, with a forced fallback answer if confidence is never reached. Notably, there is no "Instruction Agent" — the role other pipelines use to request re-captioning is replaced by pre-computed clip memory plus the router.

Why This Matters

Research impact. The paper reframes efficiency in long-video QA as an allocation problem rather than a compression problem. It challenges the assumption that text is merely a lossy substitute for video, showing that language memory can be a stronger carrier for temporal structure than dense frames, and it makes per-query visual cost an explicit design parameter that can be measured, swept, and tuned.

Real-world applications:

• On-device personal video memory — searching hours of recorded footage on a phone or laptop where storage, compute, and uplink bandwidth are limited.
• Interactive video assistants — answering questions about films, lectures, sports, or surveillance archives where latency budgets forbid dense frame processing.
• Wearable visual aids — augmented-reality glasses that need to reason about a user's surroundings without streaming continuous video to the cloud.
• Cloud video platforms and Q&A services — reducing inference cost and bandwidth per query across large video libraries.

Industry relevance. The edge-cloud split maps directly onto commercial serving architectures: cheap indexing runs once per video, and expensive multimodal reasoning is invoked only where it adds measurable value. The reported 99.1% frame reduction and the tunable accuracy-cost frontier are directly actionable for teams balancing GPU cost against answer quality. The paper also honestly notes the router adds an LLM call, so frame savings do not automatically translate to lower wall-clock latency — a distinction practitioners care about.

Future Directions

1. Better routing granularity. The router is currently a binary, zero-shot classifier and misroutes boundary cases (for example, a chronological question whose answer hinges on a specific visual change). A continuous, calibrated confidence score — trained from a small held-out set or distilled from oracle traces — could improve per-skill balance, particularly on attribute-heavy skills where the router under-allocates frames.

2. Robustness to caption and segmentation failure. If question-critical details never appear in the offline captions, text-only backtracking cannot recover them, and bounded keyframe sampling may still miss transient cues, small objects, brief on-screen text, or subtle motion. Improving the offline index and the system's ability to diagnose why text reasoning failed are open problems.

3. Egocentric and wearable video. The event segmentation relies on shot-boundary detection, which works for edited content but not for continuous first-person recordings where hard cuts are rare. Replacing it with audio-visual event detection or LLM-prompted scene-change estimation is a natural extension.

4. Real edge deployment. The edge-cloud separation is simulated, not physically deployed. Characterizing device-side compute limits, communication overhead, network latency variability, quantization, and energy consumption on representative hardware remains unmeasured.

Target Audience

Researchers and engineers working on multimodal LLMs, long-video understanding, and agentic retrieval pipelines will get the most from this paper. It is also valuable to practitioners designing edge-cloud inference systems who need concrete accuracy-versus-cost trade-off data, and to graduate students studying memory architectures for video reasoning. Readers without background in MLLM pipelines or video QA benchmarks will find the methodology section accessible but the experimental comparisons difficult to contextualize without prior exposure to Video-MME and InfiniBench.

Authors’ abstract

Long-video understanding on edge devices must reason over hours of content under tight compute and bandwidth budgets. Subsampling visual tokens loses temporal structure, while text-only video memories lose fine-grained visual attributes. We observe a visual-textual duality: language memories carry long-range temporal structure better than dense frames, while pixels remain decisive for attribute-level perception. Building on this insight, we propose Caption-once, Frames-onDemand (CFD), a budget-aware edge-cloud agentic framework. The edge runs a single offline captioning pass that builds a dual-track narrative index, an event-level story skeleton plus a clip-level micro-log, cached and reused across queries without re-captioning. At query time, a cloud-side MLLM reasons over the index in a story-first loop centered on a lightweight Visual-Need Router: a per-query gating module that triggers bounded keyframe retrieval only for perceptual questions (appearance, on-screen text, attribute disambiguation) and keeps temporal-structural questions in language space. The router turns visual access into a first-class, query-conditioned cost, capping per-query frame consumption regardless of video length. Experiments on long-video benchmarks demonstrate strong accuracy-efficiency trade-offs while substantially reducing online visual processing.