StreamingTOM: Streaming Token Compression for Efficient Video Understanding

AI summary

Overview

Research area: Efficient multimodal inference — specifically token compression and memory management for streaming video vision-language models (video LLMs).

Technical level: Advanced. The paper assumes familiarity with transformer kv-caches, prefill versus decoding phases, attention complexity, quantization, and streaming benchmark protocols.

Scope: A training-free, two-stage framework that compresses visual tokens before the LLM and bounds kv-cache memory after the LLM for strictly causal, long-horizon streaming video understanding.

What This Paper Is About

Streaming video models must process frames one at a time without seeing the future (causality), yet their key-value cache grows without bound as frames arrive (accumulation). Existing training-free methods only manage the cache after the LLM has already paid the full cost of processing every visual token, leaving the expensive prefill computation untouched. This paper introduces StreamingTOM, which attacks both bottlenecks: it discards most visual tokens before they enter the LLM and stores what remains in compressed 4-bit groups that are retrieved on demand.

Key Contributions

1. Strictly causal pre-LLM token reduction. The authors introduce a per-frame token budget G that cuts prefill complexity from O(t·N·L·d²) to O(t·G·L·d²), where N is tokens per frame, L is the number of transformer layers and d is hidden width. This is claimed as the first training-free streaming method to compress tokens before the LLM under strict causality.

2. A unified two-stage framework. StreamingTOM is the sequential composition OQM(16→4) ∘ CTR(N→G): Causal Temporal Reduction (CTR) handles pre-LLM compression and Online Quantized Memory (OQM) handles post-LLM memory. Both are tied together by a "group" abstraction — a fixed set of exactly G tokens per frame — so retrieval returns whole frames rather than fragmented tokens.

3. State-of-the-art training-free accuracy with strong efficiency. The method reports 63.8% average on offline benchmarks, 55.8%/3.7 on RVS, a 15.7× kv-cache compression ratio, 1.2× lower peak memory and 2× faster time to first token than the prior training-free state of the art.

4. Bounded memory over long horizons. On LLaVA-OV-7B, a one-hour stream's kv-cache drops from 18.8 GB to 1.2 GB, demonstrating that growth stays bounded without retraining the model.

Main Findings

• Offline long-video accuracy: StreamingTOM reaches an average of 63.8% across VideoMME, MLVU and EgoSchema, surpassing the strongest training-free baseline StreamMem (63.1) and the training-based Dispider-7B (57.9). Individual scores are 59.9 on VideoMME-Overall, 67.9 on MLVU and 63.7 on EgoSchema. VideoMME splits are 50.6 (Long), 57.8 (Medium) and 71.3 (Short).

• Streaming accuracy: On RVS, StreamingTOM scores 55.8 accuracy / 3.7 response quality on average — 58.3/3.9 on RVS-Ego and 53.2/3.5 on RVS-Movie — ahead of StreamMem (55.2/3.6), InfiniPot-V (54.6/3.5) and Flash-VStream (55.0/3.6). ReKV reports higher accuracy (59.0) but only with CPU offloading, which the authors exclude as impractical for real-time deployment.

• Compression ratio: The combined ratio of 4N/G with N=196 and G=50 yields 4×196/50 ≈ 15.7×.

• Memory and latency: At 256 frames, StreamingTOM shows 1.2× lower peak memory and 2× faster TTFT than LiveVLM. Across 16 to 512 frames at batch size 8, GPU memory stays roughly flat from 16.0 GB to 16.7 GB, TTFT rises from 0.17 s to 0.30 s, and throughput stabilizes near 20 tokens/s for long sequences.

• Overhead breakdown: For a 64-frame stream at batch size 8 with 50 tokens per frame, OQM adds 7.3 ms for kv storage, 6.9 ms for retrieval and 28.7 ms for 4-bit reconstruction, while CTR accelerates prefill by 3.6×, from 337.8 ms to 92.8 ms, giving a 0.20 s query TTFT.

• Ablation: The default 50-token, 4-bit setting gives the best balance — 59.9 VideoMME overall at 6.4% of the memory of a 196-token 16-bit baseline. 40 tokens (5.1% ratio) and 60 tokens (7.7% ratio) both degrade accuracy. 2-bit quantization halves memory further but consistently lowers accuracy (58.5 at 50 tokens, a value interpolated from 48 and 52 tokens).

• Positioning vs. offline compression: StreamingTOM outperforms offline compression methods DyCoke, VisionZip and HoliTom on VideoMME-Overall and EgoSchema, which the authors attribute to causal 0.5/0.2 fps coverage of longer temporal spans versus fixed 32-frame clips.

Methodology in Plain English

The framework runs two coordinated pipelines, both of which operate during the prefill phase — "pre-LLM" and "post-LLM" describe position in the pipeline, not inference stage.

Causal Temporal Reduction (CTR) works on the vision side. Each frame is encoded into N visual tokens, and CTR keeps only G of them. It uses a strictly causal two-frame window (current frame t and previous frame t−1), so no future information is used. For each token position it computes a cosine similarity between the two frames' features; high similarity means that location has not changed. Tokens above a threshold (set to 0.9 in experiments) go into a "static" set, the rest into a "dynamic" set. The budget G is then split between the two sets in proportion to their sizes. Dynamic tokens, which carry new information, are selected by top-k based on attention-derived saliency scores taken from the vision encoder at zero extra cost; static tokens, which are redundant, are consolidated by density-based clustering. The output is exactly G tokens per frame, which keeps per-frame latency predictable. Only the previous frame's features are held as state, so memory is O(Nd) and independent of stream length, and per-frame cost is O(N + G²).

Online Quantized Memory (OQM) works after the LLM. Each group of G tokens is quantized independently to 4-bit using a per-head, per-channel scale and offset, packed into uint8, and stored alongside a single representative key obtained by averaging the group's keys. System tokens such as instructions stay in FP16. When a query arrives, the decoder compares its state against the stored representative keys in full precision, picks the top-k most similar groups, and dequantizes only those to form the active kv-cache. Total storage scales with stream length while active memory stays bounded by O(k·G·d), with k much smaller than the total number of frames.

Experimental setup: Evaluation covers offline long-video benchmarks (VideoMME, MLVU, EgoSchema) sampled at 0.5 fps for clips under 30 minutes and 0.2 fps for longer ones, plus online streaming benchmarks (RVS-Ego, RVS-Movie) where questions arrive after their end timestamps. Experiments follow the ReKV streaming protocol on a single NVIDIA A6000 (48 GB) with FP16 mixed precision and greedy decoding. CTR retains 50 tokens per frame with a 0.9 similarity threshold; OQM uses a 12k-token budget with groups of size 50; the streaming encoder batch size defaults to 32. RVS grading uses GPT-3.5-turbo-0125 under a 28 GB GPU memory limit with no CPU offloading.

Why This Matters

Impact on research. The paper argues that post-LLM kv-cache management alone is insufficient, because prefill compute has already been spent by the time compression happens. Framing compression as a pipeline-location problem rather than an inference-stage problem, and enforcing a fixed per-frame token budget, gives the streaming community a training-free recipe for predictable latency. The group abstraction also supplies a shared unit that makes token reduction and memory management composable rather than conflicting.

Real-world applications (the application areas named in the paper):

• Autonomous driving, where frames arrive continuously and future frames are unavailable.
• Embodied AI, where perception and decision-making must run under fixed compute budgets.
• Live video assistants, where questions are asked about a stream in progress.
• Long-session video monitoring or analysis, where a one-hour stream must fit in GPU memory — the paper reports reducing 18.8 GB to 1.2 GB on LLaVA-OV-7B.

Industry relevance. The method is training-free and plug-and-play across backbones, so it can be layered onto existing video LLM deployments without retraining. The reported bounded memory growth and stable throughput (roughly 20 tokens/s for long sequences) target the operational requirements of always-on video services, where per-frame cost must be predictable rather than merely small on average.

Future Directions

• Alternative saliency and selection signals. CTR currently reuses attention-derived saliency scores from the vision encoder's final layer only; whether richer or multi-layer saliency improves selection is not explored in this paper.

• Pushing quantization below 4 bits. The ablation shows 2-bit quantization consistently hurts accuracy, leaving open how to make lower-bit storage viable without the accuracy loss the authors observe.

• Retrieval quality at scale. OQM retrieves at most k groups by cosine similarity against averaged representative keys; how this behaves when the stream is much longer than the evaluated horizons, or when many frames are similar, is not reported.

• Generalization beyond the evaluated backbones. Results are reported for LLaVA-OV-7B and the standard streaming benchmarks; the paper does not report how the fixed budget G and threshold τ_c should be re-tuned for other models or domains.

Target Audience

Researchers and engineers working on efficient multimodal inference, streaming video LLMs, and kv-cache compression. It is most useful to readers who already understand transformer attention and quantization and want a concrete, training-free design for bounding both prefill cost and cache growth. Readers looking for an introductory treatment of video-language models, or for training-based approaches, will find this specialized.

Authors’ abstract

Unlike offline processing, streaming video vision-language models face two fundamental constraints: causality and accumulation. Causality prevents access to future frames that offline methods exploit, while accumulation causes tokens to grow unbounded, creating efficiency bottlenecks. However, existing approaches only regulate post-LLM kv-cache, leaving costly pre-LLM prefill unchanged. We introduce StreamingTOM, a training-free, plug-and-play two-stage framework that addresses both pre-LLM and post-LLM bottlenecks. Causal Temporal Reduction imposes a fixed per-frame budget and selects tokens based on adjacent-frame changes and token saliency, drastically reducing per-frame prefill cost by processing only a compact subset of visual tokens, ensuring predictable latency. Online Quantized Memory stores tokens in 4-bit format, retrieves relevant groups on demand, and dequantizes them, keeping the active kv-cache bounded regardless of stream length. Experiments demonstrate our method achieves $15.7\times$ kv-cache compression ratio; compared to prior SOTA (LiveVLM), it delivers $1.2\times$ lower peak memory and $2\times$ faster TTFT. StreamingTOM achieves state-of-the-art accuracy among training-free methods with an average of $63.8\%$ on offline benchmarks and $55.8\%$ accuracy and $3.7$ score on RVS. These results demonstrate that real-time streaming video understanding with bounded active memory is achievable without model retraining.