ShallowStream: Index Shallow then Answer Deep for Streaming Video Understanding

AI summary

Overview

• Research area: Computer vision / multimodal large language models (MLLMs), specifically streaming (causal) video understanding and inference efficiency.
• Technical level: Advanced — the paper assumes familiarity with Transformer layers, KV caches, prefill computation, attention scoring, and retrieval-based context selection.
• Scope: The paper proposes ShallowStream, a two-stage framework that uses only the shallow layers of an MLLM to encode a continuous video stream and build a lightweight retrieval index, then spends full-depth computation only on the evidence selected to answer a specific query.

What This Paper Is About

Streaming video understanding requires a model to keep watching a video that never ends and answer questions that arrive unpredictably later, without knowing in advance what will be asked. The dominant cost in this setting is that every incoming frame is usually pushed through the entire depth of a multimodal language model before any question is known, which is expensive and makes the KV cache grow in proportion to that prefill depth. The paper's goal is to cut this steady-state cost by doing the streaming work with shallow layers only, and reserving deep, full-model computation for the small set of frames that a query actually needs.

Key Contributions

1. A depth-based decomposition of streaming video understanding. ShallowStream splits the workload into query-agnostic stream processing (performed by the shallow MLLM, layers [0, P)) and query-time answering (performed by the full-depth MLLM), instead of pruning tokens, merging tokens, quantizing, or offloading context as prior work does.
2. An always-on shallow visual index. Incoming video units are encoded only through shallow layers; their shallow-layer keys and values, input visual states, and timestamps are retained per unit, and a fixed-length unit descriptor is built by averaging the last shallow layer's keys over the unit's visual tokens and applying L2 normalization.
3. A query-logit routing gate. A single text-only forward pass over a fixed routing prompt with two single-token choices (A = earlier video memory needed, B = latest segment sufficient) produces a logit difference compared against a backbone-specific threshold, deciding whether retrieval runs at all — without training a separate router or consuming benchmark labels.
4. Token-vote retrieval with diversity selection and optional long-cluster compression. Shallow-layer, RoPE-aware Q-K attention from the last prompt token is averaged over heads, top tokens vote for their source units, the top 4K vote-ranked candidates undergo max-min diversity selection to K units, hits are expanded to temporal neighbors, and — under a memory budget — older units are folded into fixed-size cluster representatives via running averages.

Main Findings

• Shallow layers already retrieve well. On LVBench, Qwen3-VL-8B shows strong retrieval capability at layer 4 and LLaVA-OneVision-7B at layer 3, out of 28 and 32 total layers respectively.
• Large efficiency gains. ShallowStream reduces per-frame prefill by up to 52.1x and 10-second end-to-end latency by up to 11.9x, while achieving performance on par with the strongest existing streaming methods.
• OVO-Bench results. With Qwen3-VL-8B, ShallowStream reaches 69.5 average (Real-Time Visual Perception 80.9, Backward Tracing 58.1); the with-compression variant reaches 69.2. With LLaVA-OneVision-7B it reaches 62.2, and 62.3 with compression. For comparison in the same tables, Qwen3-VL-8B + OASIS scores 67.7, + SimpleStream 66.4, and + HERMES 57.2; LLaVA-OneVision-7B + SimpleStream scores 60.3 and + HERMES 58.1.
• StreamingBench results. On the Real-Time Visual Understanding subset, ShallowStream scores 78.2 with Qwen3-VL-8B (78.0 with compression) versus 76.6 for + HERMES, 76.5 for + SimpleStream, and 75.9 for the base Qwen3-VL-8B. With LLaVA-OneVision-7B it scores 75.5 (75.6 with compression), the highest among the LLaVA-OneVision-7B-based rows listed.
• Memory stays flat under compression. Across prefixes of 64 to 1,024 frames, long-cluster compression keeps peak GPU memory nearly flat at about 18 GiB, while the uncompressed variant grows to 21.76 GiB; HERMES and OASIS require more GPU memory at long prefixes.
• Real-time headroom is preserved either way. On five long videos containing 588–1,198 sampled frames with no frame cap, at an 80-second query interval both compression-on and compression-off variants require about 2.6 s of compute, compared with 6.2 s for HERMES and 50.4 s for OASIS.
• The calibrated gate behaves selectively. The gate activates historical retrieval mainly for retrospective (Backward) questions while keeping recent context for current-scene (Real-Time) questions, avoiding both the always-recent failure mode and indiscriminate retrieval noise.
• Token voting outperforms alternatives. Token voting from the last prompt token outperforms pooled shallow Q-K and the independent SigLIP encoder when diversity and temporal expansion are disabled; adding max-min diversity on top gives a further gain by spreading the limited evidence budget.
• Qualitative case study. On an OVO-Bench EPM question ("Where is the red and white checkered rug?", ground-truth answer E), pooled shallow Q-K repeatedly selects similar kitchen views and predicts B; max-min diversity broadens coverage but still answers B; only the final token-vote retriever with max-min diversity selects complementary views and answers E correctly.
• Boundary choice. The analyses report that a boundary of P = 5 achieves the strongest accuracy at low stream-processing cost.

Methodology in Plain English

The researchers start from an empirical check: they take a long video, let different layers of the model act as the "retriever" that ranks historical video units by relevance to a question, and then answer using only the top-ranked units. Retrieval quality is already strong very early in the network (layer 4 of 28, layer 3 of 32), so there is no reason to run the whole model on every frame.

During streaming, each arriving video unit (2 sampled frames for Qwen3-VL, 1 for LLaVA-OneVision) is passed only through the shallow layers. The keys and values produced there are stored per unit as a full-history shallow cache, along with the unit's input visual state and timestamp. A compact descriptor per unit is made by averaging the last shallow layer's keys over visual tokens and normalizing. If memory exceeds a budget, older units are merged incrementally into clusters whose centroid is compared to each new unit's descriptor, and each cluster keeps a fixed-size representative rather than all its members.

When a question arrives, the question is appended to the cached history and itself run through the shallow layers. A separate text-only pass scores two tokens (A vs B) and their logit difference, compared to a frozen threshold calibrated on a benchmark-independent set, decides whether history retrieval is invoked. If invoked, attention from the last prompt token to candidate visual tokens is reconstructed at each shallow layer, heads are averaged, the top tokens vote for their source units, the top 4K candidates go through max-min cosine-distance selection down to K units, and the survivors are expanded to their temporal neighbors. Finally, only the selected units' stored input-level visual states (plus recent context and the question) are pushed through all L language layers to generate the answer — so no unit ever receives full-depth prefill unless a query selected it.

Why This Matters

The paper reframes streaming video cost around model depth rather than only around token count or cache size: the expensive deep computation is deferred until a question identifies which evidence deserves it.

• Impact on research: It provides an alternative axis for streaming-video efficiency work, and reports an operating point that combines near-state-of-the-art accuracy with large reductions in per-frame prefill, latency, and GPU-memory growth. It also argues that using a retriever that is part of the same backbone (shallow layers) beats an independent encoder such as SigLIP in the reported ablation.
• Real-world applications (as listed by the authors):
  • Embodied intelligence and autonomous driving, where frames arrive continuously and decisions cannot wait.
  • Surveillance and early warning systems that must monitor an unbounded stream.
  • Wearable assistants and AR glasses, where compute and memory budgets are tight.
  • Industrial monitoring, sports commentary, and fitness coaching.
• Industry relevance: The reported efficiency gains (up to 52.1x lower per-frame prefill cost, up to 11.9x lower 10-second end-to-end latency, ~18 GiB flat peak GPU memory under compression) target exactly the costs that make always-on video assistants impractical on single-GPU or edge deployments. The method is training-free and applied to off-the-shelf backbones (Qwen3-VL-8B-Instruct, LLaVA-OneVision-7B), and code is released, which lowers adoption cost.

Future Directions

• How far can the shallow boundary be pushed? The paper reports P = 5 as the accuracy/cost sweet spot for the tested backbones; whether that boundary transfers to other model families, sizes, or depths is not reported.
• Robustness of the routing gate. The gate uses a threshold calibrated once on a benchmark-independent synthetic set generated with AI tools; how sensitive routing is to that calibration, to the routing prompt, and to unfamiliar question distributions is an open question.
• Cost of compression fidelity. Long-cluster representatives replace member units with averaged states; the accuracy lost on questions whose answer lies inside a compressed interval is not reported, and the paper only notes small average differences between compression-on and compression-off runs.
• Broadening the evidence. The evaluation centers on OVO-Bench and StreamingBench with two backbones at 1 FPS, plus LVBench and five long videos for the efficiency study; behavior on other modalities, frame rates, and longer or multi-query streams is left open. The paper labels itself "Work in Progress."

Target Audience

Researchers and engineers working on streaming or online video understanding, multimodal LLM inference efficiency, and KV-cache/memory management for long context. It will also interest practitioners building always-on video assistants, surveillance, robotics, or wearable applications who need to know how much latency and GPU memory they can save by spending depth only where a question demands it. Readers without background in Transformer internals and cache mechanics will find the methodology section demanding.

Authors’ abstract

Streaming video understanding is a critical capability for real-world applications, including embodied intelligence, autonomous driving, industrial monitoring, surveillance and early warning, and wearable assistants. However, processing continuous video streams with multimodal large language models (MLLMs) is computationally expensive. Existing efforts have explored reducing streaming overhead through visual token pruning, token merging, quantization, on-demand frame retrieval, and context offloading. However, most existing methods overlook the dimension of model depth. Repeatedly executing full-depth MLLM prefill over incoming frames is prohibitively expensive, incurring substantial computational overhead and causing the KV cache to grow at a rate directly proportional to the prefill depth. To address these challenges, we propose ShallowStream, a novel framework that leverages the shallow layers of an MLLM to simultaneously perform frame encoding and retrieval index building. During stream processing, ShallowStream maintains an always-on lightweight index using the KV cache of shallow layers. During query-time answering, we leverage the attention scores generated by the shallow layers to score context frames and employ a diversity-aware selection strategy to retrieve precise and comprehensive evidence. ShallowStream achieves performance on par with the strongest existing streaming methods, while reducing per-frame prefill latency and 10-second end-to-end latency by up to 52.1x and 11.9x, respectively. Our code is available at https://github.com/CURRENTF/ShallowStream.