OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models

AI summary

Overview

Research area: Efficient inference for omnimodal large language models (OmniLLMs) — specifically multimodal token compression for joint audio-video understanding.

Technical level: Intermediate. The paper assumes familiarity with transformer attention, token sequences, and multimodal LLM pipelines, but its core idea (using audio to decide what video to throw away) is intuitive.

Scope: A training-free, inference-time token compression framework that uses audio attention signals to dynamically guide video token pruning, achieving 2.5–3.4× speedup on Qwen2.5-Omni models while preserving accuracy.

What This Paper Is About

Omnimodal LLMs process video and audio together, but the combined token sequence (10,000–20,000 tokens per video) makes inference slow and memory-hungry. Prior token compression methods were built for single modalities, mostly images or video, and ignore the audio stream entirely. This paper introduces OmniZip, the first token compression framework designed specifically for the joint audio-video setting, which exploits the observation that audio tokens dominate attention and can therefore steer which video tokens are safe to discard.

Key Contributions

1. First systematic analysis of token pruning for omnimodal inputs. The authors visualize attention heatmaps across OmniLLM layers and identify three structural patterns: audio tokens consistently receive more attention than video tokens; attention is highly localized within fixed-length time windows; and attention to raw multimodal tokens decays with layer depth.

2. An audio-guided dynamic video pruning scheme. Instead of pruning all time windows at a uniform rate, OmniZip computes an "audio retention score" per window and uses it as a proxy for information density. High-retention windows (likely containing events) get light pruning; low-retention windows get aggressive pruning, while a global pruning budget stays fixed.

3. Two complementary compression modules. An audio anchor consolidation step merges non-salient audio tokens with cross-modal similarity to video, and an interleaved spatio-temporal compression (ISTC) block alternates between temporal merging (across frames) and spatial pruning (within frames) to avoid over-reducing either dimension.

4. A training-free, FlashAttention-compatible design. OmniZip reads attention scores only from the lightweight audio encoder, not from the LLM's attention matrix. This avoids the out-of-memory failures that FastV and VisionZip hit when they materialize full attention matrices on long video sequences.

Main Findings

• Large speedups with negligible accuracy loss. On Qwen2.5-Omni-7B, OmniZip achieves a 3.42× prefilling speedup and 1.42× end-to-end latency reduction at 35% token retention, while scoring 45.3 on WorldSense versus 46.8 for the full-token baseline.

• Accuracy retention near 99%. At 45% retention (a 60% FLOPs reduction), the 7B model retains 99.1% of its original average accuracy across AVUT, VideoMME, and ShortVid-Bench.

• Substantial memory savings. GPU memory drops from 35 GB to 25 GB on the 7B model — a 10 GB reduction — which matters for deployment on 48 GB consumer/prosumer GPUs.

• Audio tokens matter more than video tokens. Ablations show that audio should be pruned less aggressively than video (e.g., ρ_a = 0.3, ρ_v = 0.6 at 45% retention). Excessive pruning on either modality degrades performance sharply.

• Time-window locality is real and exploitable. Attention concentrates within time windows and decays across them, which justifies window-level (rather than global or frame-level) compression decisions.

• ISTC outperforms prior video compressors. Replacing ISTC with DyCoke's temporal merging module drops AVUT score by 0.9 and WorldSense by 0.9, confirming that handling both temporal and spatial redundancy matters.

• Global token selection fails in the omnimodal setting. VisionZip-style global selection, which picks video and audio tokens independently, disrupts cross-modal alignment and performs worse than random pruning in some configurations.

• Smaller models compress better. The 3B model tolerates aggressive pruning more gracefully than the 7B model, consistent with prior findings in video LLM compression.

Methodology in Plain English

OmniLLMs arrange inputs as a sequence of fixed-length time windows, each containing both audio and video tokens. OmniZip operates window by window in three stages.

Stage 1 — Pick the important audio tokens. The audio encoder produces an attention matrix over audio tokens. OmniZip scores each audio token by how much attention it receives from all other audio tokens, then keeps the top-scoring ones as "salient." These become anchors for downstream decisions.

Stage 2 — Consolidate the rest of the audio. The non-salient audio tokens aren't simply thrown away. OmniZip uniformly samples a few anchors and, for each, finds the non-salient token most semantically similar to the paired video segment (via cosine similarity between normalized audio and video embeddings). Those tokens get merged into the anchor, preserving context coverage.

Stage 3 — Prune video based on audio density. For each time window, OmniZip computes an audio retention rate — the fraction of audio tokens kept. A high retention rate signals an information-dense window, so video pruning there is conservative. A low rate signals a sparse window, so video pruning is aggressive. The per-window rates are then normalized so the total video pruning matches a global budget.

Video pruning itself (ISTC). Within each window, the minimum unit is four frames. Frames 2 and 4 are compared to their neighbors via cosine similarity; highly similar tokens are dropped as temporally redundant. Frames 1 and 3 go through density-peak clustering with k-nearest neighbors (DPC-KNN), which keeps tokens with high density and high distance to higher-density neighbors — a proxy for "this token represents something distinct." This alternation avoids stripping too much from either the temporal or the spatial axis.

The entire pipeline is training-free and adds less than 40 ms of overhead. Because it never inspects the LLM's internal attention matrix, it stays compatible with FlashAttention.

Why This Matters

Impact on research. This is the first paper to treat audio-video token compression as a joint problem rather than two independent single-modal problems. It establishes that cross-modal signals — specifically, audio attention — carry information about which video tokens are redundant, opening a new design axis for efficient multimodal inference. It also demonstrates a practical workaround for the FlashAttention incompatibility that plagues attention-based pruning methods.

Real-world applications:

• Real-time video assistants and meeting summarization. Long recordings with speech and visuals can be processed at a fraction of the current cost, making on-device or low-latency assistants viable.

• Autonomous driving and robotics perception. Continuous audio-visual streams from vehicles or robots can be fed to larger reasoning models within tight latency and power budgets.

• Accessibility tools. Audio descriptions, live captioning, and scene understanding for visually or hearing-impaired users benefit from the speedup without accuracy loss.

• Content moderation and media analysis. Large-scale platforms can run audio-video understanding over far more content per GPU hour, reducing infrastructure costs.

Industry relevance. The 10 GB memory reduction is the most commercially significant number. It moves OmniLLM inference from 80 GB datacenter GPUs onto 48 GB or even 24 GB hardware, which directly affects deployment cost per query. Because the method is training-free, it can be applied as a drop-in wrapper around existing checkpoints from Qwen, or similar OmniLLM families, without retraining or fine-tuning budgets.

Future Directions

1. Architectures without explicit time windows. The paper's design leans on the time-window concatenation convention used by Qwen2.5-Omni and most current OmniLLMs. If future models fuse modalities through cross-attention or learned alignment instead, the windowing assumption needs rethinking, though the authors argue temporal locality will persist.

2. Higher compression ratios for omnimodal inputs. The discussion notes that visual-only methods routinely reach 70–85% reduction, while omnimodal settings appear harder — audio-video tasks are more sensitive to pruning. Closing this gap is an open problem.

3. Task-adaptive pruning ratios. The ablation hints that optimal ρ_a and ρ_v depend on whether a task is audio-dominated or video-dominated. Automatically inferring this per-task, without a validation set, is a natural next step.

4. Joint compression with the language model's own attention. Since OmniZip deliberately avoids reading the LLM's attention matrix for FlashAttention compatibility, there may be room for a lightweight approximate signal from the LLM that adds discriminative power without breaking memory efficiency.

5. Extension to other modality combinations. The same "dominant modality guides pruning of the other" principle could apply to video-text, audio-text, or three-modality settings.

Target Audience

This paper is most valuable to researchers and engineers working on efficient multimodal inference — particularly those deploying video LLMs or OmniLLMs in production and hitting memory or latency walls. It will also interest multimodal architecture researchers studying cross-modal attention patterns, and graduate students entering the token compression subfield, since it provides a clean introduction to attention-based pruning plus a novel cross-modal extension. Practitioners who need to run Qwen2.5-Omni or similar models on constrained hardware will find the memory and speedup numbers directly actionable.

Authors’ abstract

Omnimodal large language models (OmniLLMs) have attracted increasing research attention of late towards unified audio-video understanding. However, the high computational cost of processing longer joint audio-video token sequences has become a key bottleneck. Existing token compression methods have not addressed the emerging need to jointly compress multimodal tokens. To bridge this gap, we present OmniZip, a training-free, audio-guided audio-visual token-compression framework that optimizes multimodal token representation and accelerates model inference. Specifically, OmniZip first identifies salient audio tokens, then computes an audio retention score for each time group to capture information density, thereby dynamically guiding video token pruning and preserving cues from audio anchors enhanced by cross-modal similarity. For each time window, OmniZip compresses the video tokens using an interleaved spatio-temporal scheme. Extensive results demonstrate the merits of OmniZip: it achieves a 3.42X inference speedup and a 1.4X memory reduction over other top-performing counterparts, while maintaining the performance of OmniLLMs without training.