Elephant in the Fridge: Constant-Memory Frame Packing for Long Video Understanding

AI summary

Overview

• Research area: Computer vision and multimodal learning, specifically vision-language models (VLMs) applied to long-video and streaming-video understanding.
• Technical level: Intermediate. The paper assumes familiarity with VLMs, visual tokenization, and CLIP-style encoders, but its central idea (fixed-budget frame packing) is conceptually simple.
• Scope: The paper introduces FramePack-U, a training-free inference-time frame packing method that compresses arbitrarily long videos into a constant-memory visual representation for off-the-shelf VLMs, and evaluates it on LongVideoBench, MLVU, and Video-MME across several Qwen backbones.

What This Paper Is About

Vision-language models typically encode every video frame into visual tokens, so the token sequence grows linearly with video length and memory cost and latency grow without bound. Prior fixes such as token reduction and keyframe selection shorten the sequence but leave this scaling intact, and they force lossy binary keep-or-discard decisions. FramePack-U instead compresses videos of any length into a fixed-size set of visual tokens at constant, O(1) memory cost, without fine-tuning or modifying the underlying model.

Key Contributions

1. A constant-memory frame packing framework (FramePack-U). It extracts a fixed-size visual representation from videos of arbitrary length, achieving O(1) memory complexity, and plugs directly into off-the-shelf VLMs at inference time with no fine-tuning or architectural modification.
2. A content-aware compression strategy for understanding. Departing from the original FramePack formulation (which favors temporal continuity and proximity for video generation), FramePack-U scores each frame by its semantic similarity to the text prompt via CLIP and ranks frames in a diversity-driven greedy manner while retaining full temporal coverage.
3. Hierarchical resolution packing under a fixed token budget. Selected frames are assigned heterogeneous spatial downsampling factors (d = 1, 2, 4, 8, ...) so that top-ranked frames keep finer resolution and lower-ranked frames are progressively compressed, with total cost bounded by M Σ f_d/d² ≤ B.
4. An empirical instantiation and extensive benchmarking. The paper searches over mixed-resolution schemes and identifies configurations that generalize across architectures, parameter scales, and frame budgets, reporting consistent gains on LongVideoBench, MLVU, and Video-MME against vanilla sampling and against FastVID, VisionZip, and FrameFusion.

Main Findings

• Large gains on the MoE backbone. On Qwen3.5-35B-A3B with a 128-frame budget, FramePack-U yields +12.9%, +10.6%, and +9.7% over the official model on LongVideoBench, MLVU, and Video-MME respectively (72.3 vs. 59.4, 82.4 vs. 71.8, 76.4 vs. 66.7).
• Consistent gains on the dense backbone. On Qwen3.6-27B with a 128-frame budget, gains are 6.4%, 8.0%, and 2.7% (74.4 vs. 68.0, 83.9 vs. 75.9, 76.3 vs. 73.6).
• Gains persist at larger budgets. On Qwen3.5-35B-A3B at 512 frames: 73.0 (+13.2), 84.7 (+10.4), 75.8 (+8.5). On Qwen3.6-27B at 512 frames: 74.9 (+3.1), 82.1 (+4.1), 76.3 (+1.9).
• Token reduction and keyframe selection baselines lose accuracy rather than gain it. At a 128-frame budget on Qwen3.6-27B, FastVID scores 59.9 (-8.1), 67.3 (-8.6), 61.0 (-12.6); VisionZip 57.7 (-10.3), 67.4 (-8.5), 63.2 (-10.4); FrameFusion 57.8 (-10.2), 65.4 (-10.5), 63.5 (-10.1), all relative to Vanilla.
• Results are competitive with much larger proprietary models. FramePack-U on Qwen3.6-27B at 128 frames (74.4 / 83.9 / 76.3) is compared against GPT-5 (72.6 / 77.7 / 83.3), GPT-5 mini (69.7 / 69.1 / 77.3), Gemini 3 Pro (75.9 / 75.7 / 88.6), Gemini 2.5 Flash (73.1 / 75.1 / 84.2), and Claude Sonnet 4.5 (65.1 / 64.0 / 74.2).
• Robustness to the packing scheme. Under a 512-frame budget on Qwen3.6-27B, all mixed-resolution schemes tested beat the vanilla baseline (71.8 / 78.0 / 74.4): d1f128d2f1536 with 1664 frames (74.9 / 82.1 / 76.3), d1f256d2f1024 with 1280 frames (74.7 / 82.2 / 75.9), d1f384d2f512 with 896 frames (74.6 / 82.9 / 76.0), and d1f480d2f128 with 608 frames (74.3 / 81.0 / 76.5). No single scheme is universally best across benchmarks.
• Relevance and diversity are both necessary. Removing prompt relevance (Diversity-only) drops Video-MME by 6.5, 6.3, and 6.1 points under the three 512-frame schemes. Ranking by query-frame similarity alone (Relevance-only) also degrades results, e.g. 74.0 (-0.9) and 73.1 (-1.5) on LongVideoBench versus 74.9 and 74.6 for the full method.
• Benefits scale with model capacity. At 512 frames with d1f256d2f1024, gains over vanilla were +0.9 / -1.0 / -1.2 for Qwen3.5-4B, +1.6 / -0.6 / -0.4 for Qwen3.5-9B, and +2.5 / +4.2 / +1.5 for Qwen3.6-27B. Smaller VLMs sometimes degrade because they cannot integrate many visual observations.
• Preprocessing overhead is small and constant. The CLIP-based scoring stage adds 0.169 to 0.183 s of preprocess latency for 15s videos and 4.366 to 4.729 s for 3600s videos; preprocess memory is reported as the introduced CLIP-based ranking stage. FramePack-U reduced average VLM inference latency on Qwen3.5-9B from 29.877 s to 26.048 s and average overall latency from 29.877 s to 26.664 s on LongVideoBench.
• A caveat on the largest model's latency. For Qwen3.6-27B the average overall latency with FramePack-U (64.313 s) is slightly above vanilla (64.473 s in VLM inference; overall 64.313 vs. 64.473 table values), which the authors attribute to I/O: the vanilla model was restricted to at most 224 frames due to GPU memory constraints, while FramePack-U decodes candidate frames at fps=1.0, up to 3600 frames for a 3600s video.
• Denser feature-space coverage. A t-SNE comparison uses FramePack-U with n=848 under scheme d1f16d2f320d4f512 versus Uniform Sampling with n=128 at full resolution, showing denser coverage of the video's visual feature distribution under the same memory budget.

Methodology in Plain English

The method takes a long video and a text question, and produces a fixed-size bundle of visual tokens for a VLM to read. It works in two stages.

First, it scores every candidate frame against the text prompt using CLIP cosine similarity. Starting from the highest-scoring frame, it repeatedly picks the next frame that best balances prompt relevance against visual dissimilarity from frames already chosen, controlled by a trade-off coefficient λ (set to 0.7). The greedy procedure yields a single ranked ordering of N frames; frames chosen earlier are considered higher priority. A query-agnostic alternative, a binary-tree midpoint ordering that recursively ranks frames by temporal midpoints, is used as a baseline.

Second, it packs those ranked frames into a fixed token budget by assigning each a spatial downsampling factor. The top-ranked frames are kept at full resolution (d = 1), and lower-ranked frames are progressively downsampled (d = 2, 4, ...), since downsampling by d in each dimension yields M/d² tokens per frame. Because the budget B is fixed, the scheme can in principle retain all frames of a video of any length; N is treated as a tunable parameter trading temporal density against per-frame fidelity. The packed frames are then sorted back into temporal order and passed to the VLM.

Experiments were run with CLIP-ViT-B/32 for scoring and ViT global features for frame-to-frame similarity. All frames were sampled offline for benchmarking (an online mode is noted as possible for lower latency). Resolution settings followed the defaults for LongVideoBench and Video-MME; for MLVU, candidate frames were proportionally resized so the longer edge is 1280 pixels. Runtime evaluations used a single NVIDIA A800 GPU.

Why This Matters

The paper targets a structural bottleneck rather than a benchmark-specific trick: making visual memory cost independent of video duration, without retraining the model. That reframes long-video understanding as a budget-allocation problem rather than a sequence-shortening problem, and it lets weaker or older VLMs gain substantially without architectural changes.

Real-world applications named or implied in the paper:

• Surveillance analysis, where feeds are continuous and unbounded.
• Autonomous driving, where incoming frames arrive as a stream that must be processed continuously.
• Sports commentary and movie comprehension, which involve hour-long content with evolving narratives and long-horizon causal dependencies.
• Instructional or educational assistants, which need to reference material presented across a long session.

Industry relevance: the method is training-free and model-agnostic, so it can be dropped into existing VLM inference stacks as a preprocessing step. The reported ability to keep memory constant while packing far more frames (608 to 1664 frames under a 512-frame budget) speaks directly to serving cost and to running long-video workloads on memory-constrained hardware.

Future Directions

• Choosing N automatically. The paper shows performance is not monotonically increasing in the number of packed frames, and that the optimum depends on model capacity and pre-training data. A principled selection rule for N remains open.
• Adapting to weaker models. Gains were unstable or negative for Qwen3.5-4B and Qwen3.5-9B under large budgets, so how to pack effectively for lower-capacity VLMs is unresolved.
• Online and streaming operation. The authors note frames were sampled offline for benchmarking and that an online setting would reduce latency; a streaming evaluation is a natural next step.
• Remaining limitations. The Limitations section of the provided content is truncated after stating that the benefit of frame packing depends on the capacity of the underlying VLM; the additional limitations the authors list are not reported here.

Target Audience

Researchers and engineers working on vision-language models, long-video and streaming video understanding, or efficient multimodal inference. It is also useful for practitioners who want a training-free preprocessing layer to cut memory and latency in deployed video VLM systems, and for readers interested in how fixed-budget context packing, originally developed for video generation, transfers to understanding tasks.

Authors’ abstract

Hour-long videos, streaming feeds, and movie-length content are pushing vision-language models (VLMs) past their breaking point -- the standard recipe of encoding each frame into visual tokens yields sequences that grow linearly with video length, causing prohibitive memory consumption and inference latency. Existing approaches, such as token reduction and keyframe selection, shorten the token sequence but leave the O(n) scaling intact, failing to resolve the fundamental bottleneck. We propose FramePack-U, a strikingly simple, training-free frame packing framework that compresses arbitrarily long videos into a fixed size visual representation at constant memory cost. FramePack-U requires no fine-tuning or architectural modification -- it plugs directly into off-the-shelf VLMs at inference time. Inspired by FramePack from the generation literature, we propose a content-aware compression strategy for visual understanding under a fixed token budget. Specifically, FramePack-U (1) scores each frame by its semantic similarity to the text prompt and ranks frames in a diversity-driven manner, and (2) packs the selected frames at varying resolutions into a fixed-length context window. This concentrates the memory budget on frames that are both instruction-relevant and visually diverse. While FramePack-U can in theory pack arbitrarily many frames into the fixed budget, we identify an empirical instantiation that generalizes robustly across settings. Extensive experiments on LongVideoBench, MLVU, and Video-MME show that FramePack-U consistently outperforms full-resolution baselines and prior token reduction and keyframe selection methods across diverse model families, parameter scales, and frame budgets. Notably, on Qwen3.5-35B-A3B with a 128-frame budget, FramePack-U yields gains of +12.9\%, +10.6\%, and +9.7\% over the official model on the three benchmarks, respectively.