OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Models

AI summary

Overview

Research area: Efficient inference for omni-modal large language models (joint audio-video-language understanding), specifically token compression.

Technical level: Intermediate. The paper assumes familiarity with transformer token sequences, cross-attention, and multimodal LLM architectures, but its central idea is intuitive and can be followed without deep background.

Scope: The paper proposes OmniSIFT, a two-stage compression framework that prunes redundant video tokens first and then uses the surviving visual tokens to decide which audio tokens to keep, reducing omni-modal input length to 25–35% of the original while matching or beating full-token performance on several benchmarks.

What This Paper Is About

Omni-modal LLMs must process densely sampled video frames and high-resolution audio streams simultaneously, and a 20-second clip can produce over 20,000 tokens. This makes inference slow and memory-intensive. Existing token compression methods were designed for vision-only models, and the few attempts at omni-modal compression either treat audio and video as equally important or compress each modality independently, ignoring the fact that audio saliency usually depends on visual context. The goal of this paper is a compression scheme that exploits this real-world asymmetry: video redundancy can be judged from the video itself, while whether a sound matters often only becomes clear once you know what is on screen.

Key Contributions

1. Design principles for omni-modal compression. The authors argue, from the perceptual asymmetry of human audio-visual processing, that effective compression should be modality-asymmetric and vision-guided, lightweight, and compatible with efficient attention operators such as FlashAttention.

2. The OmniSIFT framework. A two-stage pipeline consisting of Spatio-Temporal Video Pruning (STVP), which removes spatial and temporal redundancy from visual tokens within each 2-second multimodal chunk, and a Vision-Guided Audio Selector (VGAS), which uses the pruned visual tokens to score and select audio tokens.

3. End-to-end trainable compression. VGAS is a lightweight cross-attention module plus a two-layer MLP scorer, optimized through a straight-through estimator so gradients flow through the discrete top-k selection. The whole addition costs only 4.85M parameters on Qwen2.5-Omni-7B.

4. Broad empirical validation. Experiments on five audio-visual benchmarks (VideoMME with audio, DailyOmni, WorldSense, OmniVideoBench, video-SALMONN-2) across two model scales (7B and 3B) and two retention ratios (35% and 25%), plus structural, paradigm-level, and ratio-sweep ablations.

Main Findings

• Consistent wins over compression baselines. At both 35% and 25% retention, OmniSIFT achieves the highest accuracy among all compression methods on every benchmark tested, for both the 7B and 3B backbones.

• Matches or exceeds full-token models on several tasks. With 35% of tokens on Qwen2.5-Omni-7B, OmniSIFT scores 50.0 on WorldSense versus 49.7 for the full-token model, and 68.3 average on VideoMME versus 67.6. At 25% retention it still edges past full tokens on WorldSense (49.9 vs 49.7) and DailyOmni (72.5 vs 72.2). The authors attribute this to the removal of noise-carrying redundant tokens, not just to efficiency.

• Large efficiency gains at near-zero latency cost. On Qwen2.5-Omni-7B, OmniSIFT cuts peak GPU memory from 27.59 GB to 22.91 GB and total inference time from 15,097 s to 8,756 s, with end-to-end latency dropping from 4.94 s to 2.86 s. Despite adding a learned cross-modal module, its latency and memory are on par with training-free baselines like OmniZip and DyCoke.

• Visual guidance is essential for audio selection. Replacing VGAS's cross-attention with audio-only self-attention drops accuracy by 3.9% on DailyOmni and 2.9% on WorldSense, showing that audio token importance cannot be reliably judged from the audio stream alone.

• Asymmetry beats symmetry, especially under aggressive compression. Against a retrained, OmniZip-style modality-symmetric baseline, OmniSIFT leads by 2.7 points on DailyOmni at 35% retention, and the gap widens to 3.7 points at 25%, where symmetric compression degrades faster.

• Robustness to extreme audio compression. As the audio compression ratio rises from 0.3 to 0.9, OmniZip's accuracy falls from roughly 48.9% to 44.0%, while OmniSIFT stays above 49.3% across the entire range.

• Fine-grained reasoning holds up better. On DailyOmni at 25% retention with the 7B model, OmniSIFT scores 66.7 on Event Sequence and 68.9 on AV Event Alignment, versus 61.8 and 59.7 for OmniZip — categories that require intricate temporal and cross-modal reasoning.

• A concrete failure mode of symmetric compression. In an OmniVideoBench case study, a low audio saliency score during a scoreboard change causes OmniZip to strip the scoreboard patches entirely, producing a wrong answer. OmniSIFT retains those visually salient patches and answers correctly.

Methodology in Plain English

The framework operates inside the temporal chunks that omni-modal models already use to align the two modalities. In Qwen2.5-Omni, each chunk covers 2 seconds and contains two consecutive frames plus the audio for that window.

Stage one — pruning video. The first frame of a chunk establishes the scene layout. The method averages all its patch tokens into one global vector, then scores each patch by how far it deviates from that average using cosine distance; unusual patches score high. The second frame is handled differently: each of its patches is compared to the corresponding patch in the first frame, so patches that change over time (motion, new objects) score high. Within each frame, the top-scoring patches are kept according to a fixed retention budget, and the rest are discarded.

Stage two — selecting audio. The surviving video tokens serve as keys and values in a small multi-head cross-attention layer, with audio tokens acting as queries. This lets each audio token "look at" the visual context it accompanies. A residual connection adds the original audio embedding back in, so the acoustics are not overwritten by visual information. A two-layer MLP with a sigmoid then produces a single saliency score per audio token, and top-k selection keeps the highest-scoring ones.

Training. Because picking top-k tokens is not differentiable, the authors use a straight-through estimator: in the forward pass a binary keep/discard mask is applied exactly, while in the backward pass gradients are passed straight through to the saliency scores as if the selection had been identity. Only the VGAS module and the LLM decoder are fine-tuned, on 107K synchronized audio-visual captioning pairs from the AVoCaDO dataset, keeping the training cost modest.

Why This Matters

Research impact. The paper challenges the default assumption in omni-modal compression that audio and video deserve symmetric treatment, and it shows empirically that asymmetric, vision-guided pruning is both more accurate and more robust at aggressive compression ratios. It also provides a rare positive result where compression improves accuracy over the uncompressed model, suggesting that redundant multimodal tokens act as noise. The design principles it distills (asymmetry, lightness, operator compatibility) give future work a clear reference point, and the finding that learning a small module does not cost extra latency relative to training-free methods removes a common objection to learned compression.

Real-world applications:

• Real-time conversational assistants that reason over live audio and video, where end-to-end latency below a few seconds is a hard requirement.
• Long-video search and content moderation on platforms handling millions of hours of footage, where per-clip inference cost dominates operating budgets.
• Accessibility tools that describe video content with sound for blind and low-vision users, running on consumer hardware.
• On-device or edge robotics and AR systems that must fuse visual and audio streams under tight memory constraints.

Industry relevance. Serving omni-modal models is expensive, and the paper reports over 40% reductions in inference time and more than 4.6 GB of peak memory savings on a 7B model. That translates directly into lower serving cost and higher throughput. The involvement of the Kling team at Kuaishou indicates the technique targets production video platforms, where long-form video understanding is a core workload rather than a research curiosity. The compatibility requirement with FlashAttention also matters practically, since many attention-score-based pruning methods force a fallback to slower attention kernels.

Future Directions

• Cross-chunk temporal modeling. STVP currently measures temporal redundancy only within each 2-second chunk, so redundancy and long-range structure spanning many chunks remain unaddressed. Extending pruning to the full video timeline is the most direct open problem.

• Adaptive, query-guided budgets. The current method uses a fixed, query-agnostic token budget. Videos with non-uniform information density (a static interview versus a fast-cut action sequence) would likely benefit from dynamically allocating tokens per chunk, and from letting the user's question steer which tokens are kept.

• Fine-grained audio-visual alignment. The authors acknowledge that tasks depending on highly localized synchrony are sensitive to pruning, so better preservation of precise temporal correspondence between a sound and the moment it occurs is needed.

• Beyond single-turn QA. Extending the framework to multi-turn dialogue and audio-triggered visual evidence requires compression that respects conversational context rather than treating each input independently.

Target Audience

This paper is most useful to machine learning researchers and engineers working on efficient multimodal inference, particularly those building or deploying omni-modal and video-language models. Practitioners concerned with serving cost, latency, and GPU memory for audio-video workloads will find the efficiency numbers and paradigm comparison directly actionable. The modality-asymmetry argument is also of interest to researchers studying cross-modal representation and human-inspired model design. Readers need only a working understanding of transformer attention and multimodal tokenization; the mathematical detail is contained and the central intuition is accessible to graduate students entering the field.

Authors’ abstract

Omni-modal Large Language Models (Omni-LLMs) have demonstrated strong capabilities in audio-video understanding tasks. However, their reliance on long multimodal token sequences leads to substantial computational overhead. Despite this challenge, token compression methods designed for Omni-LLMs remain limited. To bridge this gap, we propose OmniSIFT (Omni-modal Spatio-temporal Informed Fine-grained Token compression), a modality-asymmetric token compression framework tailored for Omni-LLMs. Specifically, OmniSIFT adopts a two-stage compression strategy: (i) a spatio-temporal video pruning module that removes video redundancy arising from both intra-frame structure and inter-frame overlap, and (ii) a vision-guided audio selection module that filters audio tokens. The entire framework is optimized end-to-end via a differentiable straight-through estimator. Extensive experiments on five representative benchmarks demonstrate the efficacy and robustness of OmniSIFT. Notably, for Qwen2.5-Omni-7B, OmniSIFT introduces only 4.85M parameters while maintaining lower latency than training-free baselines such as OmniZip. With merely 25% of the original token context, OmniSIFT consistently outperforms all compression baselines and even surpasses the performance of the full-token model on several tasks.