FluxMem: Adaptive Hierarchical Memory for Streaming Video Understanding

AI summary

Overview

• Research area: Computer vision / multimodal large language models, specifically streaming (online) video understanding and visual token compression.
• Technical level: Intermediate. The paper assumes familiarity with MLLM architectures (vision encoder, projector, LLM), token pruning/merging, and KV-cache management, but its two core modules are described in self-contained algorithmic terms.
• One-sentence scope: FluxMem is a training-free, hierarchically organized memory framework that progressively compresses visual tokens across short-, mid-, and long-term buffers using two lightweight modules and Otsu-derived adaptive thresholds, evaluated on five online and offline video benchmarks.

What This Paper Is About

Streaming video applications such as robotic manipulation, autonomous driving, and smart glasses require models to process frames as they arrive and answer queries in real time, but existing MLLMs are built for pre-loaded offline video. Prior work either manages the KV cache, applies query-guided filtering, or compresses visual tokens — yet existing token compression methods such as TimeChat-Online apply a single pruning or merging policy uniformly across the whole stream. FluxMem's goal is to make memory utility time-dependent: retain recent frames densely for grounding while compressing distant history more aggressively, without any manual tuning or fine-tuning.

Key Contributions

1. A training-free hierarchical memory with two lightweight adaptive modules, giving MLLMs coherent short- and long-term video modeling for both online and offline settings.
2. Two spatiotemporal reduction modules: Temporal Adjacency Selection (TAS) at the short-to-mid boundary, which removes redundant tokens across adjacent frames, and Spatial Domain Consolidation (SDC) at the mid-to-long boundary, which merges spatially repetitive regions into compact anchors.
3. A distribution-adaptive thresholding scheme that derives per-frame thresholds from Otsu's method over similarity-score distributions, eliminating manual hyperparameter tuning and outperforming fixed-rule compression.
4. State-of-the-art results across online and offline benchmarks while discarding 60–70% of visual tokens and reducing latency and GPU memory usage.

Main Findings

• Online results: FluxMem reaches 76.4 on StreamingBench real-time (up from the 73.9 Qwen2.5-VL baseline, +2.5) and 67.2 on OVO-Bench real-time (up from 63.3, +3.5), while compressing roughly 70% of visual tokens. On OVO-Bench overall, it improves from 49.8 to 53.3 (+3.5).
• Efficiency on OVO-Bench: latency drops by 69.9% and peak GPU memory by 34.5% (812 ms and 23.5 GB versus the 2701 ms and 35.8 GB baseline), while accuracy improves by +3.5.
• Efficiency on MLVU: latency drops by 44.3% and peak memory by 31.2% (2014 ms and 28.4 GB versus 3614 ms and 41.3 GB), with +5.2 accuracy.
• Low online overhead: the per-frame update costs only 4.1 ms in total — 1.3 ms for TAS, 2.4 ms for SDC, and 0.4 ms for other operations.
• Offline performance from an online design: FluxMem reaches 65.3 on VideoMME (vs. 63.3), 73.1 on MLVU (vs. 67.9), and 61.1 on LongVideoBench (vs. 60.7), using about 65% fewer visual tokens on MLVU.
• Subtask gains: Prospective Reasoning improves by +6.5 and Spatial Understanding by +3.3 on StreamingBench; Action Recognition by +6.4 and Object Recognition by +5.9 on OVO-Bench.
• VideoMME length breakdown: short context rises from 73.8 to 76.9 and medium from 62.4 to 65.1, while long context stays stable (53.8 to 54.0) even under a token reduction regime approaching 90%.
• Hierarchical memory ablation (MLVU / VideoMME / StreamingBench / average): mid+long memory reaches 73.1 MLVU at a 65.6% drop ratio, beating either alone; on StreamingBench, short+long reaches 77.0 versus 73.9 for short only and 75.9 for long only; the full S+M+L hierarchy gives the highest average of 71.6 at 64.3% token reduction.
• Adaptive vs. fixed thresholds: in mid-term memory, the adaptive threshold attains 65.6 at a 42.8% drop ratio while the best fixed threshold obtains 65.5 at only 29.4%; in long-term memory, the adaptive threshold matches 65.4 at an 84.5% drop ratio versus 65.6 for the best fixed threshold at 63.8% compression.
• Comparison with other reduction strategies: against FIFO, Uniform, Random, and DTD, FluxMem achieves 73.1 on MLVU at a 64% drop ratio and still maintains 70.1 at an 85% token drop.
• Training-free versus SFT: FluxMem alone gives OVO-Bench overall 53.3 (+3.5) and StreamingBench real-time 76.4 (+2.5); adding supervised fine-tuning on a subset of TimeChat-Online-139K and VideoChatOnline-IT raises these to 61.4 (+11.6) and 76.7 (+2.8).

Methodology in Plain English

FluxMem treats visual memory as three cascaded buckets. Every incoming frame is encoded into tokens and pushed into short-term memory, where nothing is dropped. When short-term memory overflows (in experiments, after 8 frames), the evicted tokens pass through Temporal Adjacency Selection, which asks whether each token differs from what was at the same place in the previous frame or the next frame. Comparison uses cosine distance within a 3×3 spatial window, making it robust to small motion and camera jitter without needing optical flow. A token survives if it is novel relative to either neighbor, and survivors go into mid-term memory.

When mid-term memory overflows (after 64 frames in the online setting), the oldest tokens pass through Spatial Domain Consolidation, which builds a sparse 8-connected graph over only the tokens TAS already kept, joins nearby similar tokens with a union-find pass, and replaces each connected component with its mean "anchor." These anchors go to long-term memory, and the oldest entries there are evicted when capacity is reached.

The distinctive step is how the keep-or-drop cutoffs are chosen. Rather than a manually tuned ratio or a fixed similarity threshold, FluxMem runs Otsu's method — a classic non-parametric algorithm that maximizes inter-class variance — at runtime on each distribution of scores: temporal similarity scores for TAS and pairwise spatial distances for SDC. This lets compression tighten in high-motion scenes and loosen in static ones. The same backward-looking TAS scores double as a zero-cost trigger for proactive responses: when the fraction of tokens exceeding the adaptive threshold passes a tunable γ, the model declares a scene switch and responds without waiting for a query. At query time, tokens from all three levels are concatenated in spatiotemporal order and fed to the LLM. Everything runs in a single causal pass with O(HW) cost per overflow event.

Experiments use Qwen2.5-VL-7B on 8× A100 GPUs, with 1 fps sampling, up to 256 visual tokens per frame and 256 frames per video online, and 64 tokens per frame with a maximum sequence length of 1024 frames offline.

Why This Matters

Impact on research. The paper challenges the prevailing assumption that a single compression policy should apply uniformly across a video stream, and it demonstrates that adaptive, data-driven thresholds beat manually tuned ones. It also shows a purely online, causal framework transferring to offline long-video benchmarks, suggesting hierarchical memory may be a general-purpose mechanism rather than a streaming-specific patch.

Real-world applications.

• Robotic manipulation, where a robot must react to what it currently sees while still remembering earlier stages of a task.
• Autonomous driving, where scene changes must trigger timely responses without waiting for an explicit query.
• Smart glasses and wearable assistants, where memory and battery budgets are tight and frames arrive continuously.
• Real-time video surveillance or monitoring, where long histories must be searchable at low latency.

Industry relevance. The reported reductions (69.9% latency and 34.5% peak GPU memory on OVO-Bench, 44.3% and 31.2% on MLVU) map directly onto serving cost for always-on video inference. Because FluxMem is training-free and plug-and-play with a pre-trained MLLM, it avoids the fine-tuning and deployment overhead that training-based online models such as Flash-VStream require. The per-frame overhead of 4.1 ms is small enough to fit inside a real-time budget.

Future Directions

• Scaling the underlying model. All experiments use Qwen2.5-VL-7B; whether the hierarchical memory behaves the same way at larger or smaller model scales is not reported.
• Combining with fine-tuning. The SFT result (61.4 on OVO-Bench overall versus 53.3 training-free) suggests the training-free and fine-tuned regimes are complementary, leaving open how best to combine them.
• Tuning the trigger parameter. The proactive trigger depends on γ ∈ [0,1], a tunable sensitivity knob; the paper does not report a systematic study of how γ should be set across domains.
• Extending beyond video compression. The Otsu-based adaptive thresholding is applied only to temporal and spatial token similarity distributions, raising the question of whether the same data-driven principle transfers to other modalities or to KV-cache management, which the paper deliberately avoids.

Target Audience

Researchers and engineers working on multimodal large language models, streaming or online video understanding, and inference efficiency. It is most useful to readers who already understand how visual tokens flow through an MLLM and want a concrete, training-free recipe for reducing that flow under real-time constraints — and to practitioners deciding whether token compression, KV-cache management, or query-guided filtering is the right lever for a deployment budget.

Authors’ abstract

This paper presents FluxMem, a training-free framework for efficient streaming video understanding. FluxMem adaptively compresses redundant visual memory through a hierarchical, two-stage design: (1) a Temporal Adjacency Selection (TAS) module removes redundant visual tokens across adjacent frames, and (2) a Spatial Domain Consolidation (SDC) module further merges spatially repetitive regions within each frame into compact representations. To adapt effectively to dynamic scenes, we introduce a self-adaptive token compression mechanism in both TAS and SDC, which automatically determines the compression rate based on intrinsic scene statistics rather than manual tuning. Extensive experiments demonstrate that FluxMem achieves new state-of-the-art results on existing online video benchmarks, reaching 76.4 on StreamingBench and 67.2 on OVO-Bench under real-time settings, while reducing latency by 69.9% and peak GPU memory by 34.5% on OVO-Bench. Furthermore, it maintains strong offline performance, achieving 73.1 on MLVU while using 65% fewer visual tokens.