PACE: Pareto-Adaptive Compression for Efficient Native MLLMs

AI summary

Overview

Research area: Computer vision / multimodal large language models — specifically visual token compression for "native" (ViT-free) MLLMs.

Technical level: Advanced. The paper assumes familiarity with Transformer attention, rotary position encoding (RoPE), KV caching, FLOPs accounting, and benchmark conventions in multimodal evaluation.

Scope: The paper proposes PACE, a training-free-at-inference visual token compression framework tailored to native MLLMs with multi-axis positional encoding, built on a probing analysis of how redundancy is distributed across depth and across spatial height/width axes.

What This Paper Is About

Native MLLMs such as Emu3, Chameleon, SOLO, SAIL and NEO tokenize every pixel patch directly instead of relying on an external vision encoder, so their visual sequences are intuitively expected to be highly redundant — yet no compression method had been designed for them, because existing techniques (FastV, PyramidDrop, SparseVLM, D²Pruner and others) all assume a modular ViT-based architecture. The authors first run a pilot study on MMBench-DEV-EN comparing Qwen3-VL against a native MLLM they build, and find that ViT-based models degrade under compression while native models tolerate aggressive compression near-losslessly or even improve. The goal is to close that gap with a compression framework that exploits the structure native backbones already have, namely the dimensional decoupling of Native-RoPE.

Key Contributions

1. A probing analysis pipeline for native backbones. Two quantitative metrics are introduced — Structural Discriminability (centered effective rank of the visual token matrix divided by token count) and Saliency Lift (relative gain in per-token 3D-decomposed attention importance from the full set to the PACE-selected subset) — plus a measure of cross-dimensional heterogeneous redundancy along the height and width axes.

2. Two empirical findings that motivate the design. (i) An Initial Fusion Buffer (IFB) in the earliest layers keeps discriminability low, after which it rises monotonically with depth, forming a depth-progressive ramp that justifies conservative early pruning and aggressive deep pruning. (ii) Redundancy is distributed independently and heterogeneously along height and width, so a single conflated attention score cannot distinguish, for example, horizontal text lines (salient along height, uniform along width) from vertical structures.

3. Spatially Decoupled Importance Estimation (SDIE). Exploiting Native-RoPE's split of queries/keys into temporal, height and width components, attention is decomposed into directional saliency scores fused as (ReLU(s_i^H) + ReLU(s_i^W)) × ReLU(s_i^T) — additive over the two spatial axes, multiplicatively gated by the temporal term — with zero additional parameters since scores come from QK products already computed in the forward pass.

4. Pareto Adaptive Compression Scheduling (PACS) plus a full four-stage training study. Token budget allocation is cast as a Pareto optimization over performance preservation, efficiency and spatial coverage variance (on a 4×4 grid), jointly selecting compression depths and per-stage retention ratios; the authors also train a 2B-parameter native MLLM from scratch through PT/MT/SFT/RL because no concurrent native MLLM open-sources all four stages.

Main Findings

• Native MLLMs tolerate compression far better than ViT-based ones. On the authors' Ours-2B backbone, compression improves results at 25% retention on MMB (+1.4%, 70.7 vs 69.7) and AI2D (+2.8%, 83.1 vs 80.8), and gains grow with retention (+3.2% MMB and +5.1% AI2D at 75%). Qwen3-VL instead collapses by up to 43.7% on MathVista at 25% retention, and Intern3-VL drops 26.5% on MathVista at 25%.

• Some benchmarks remain sensitive. At 25% retention Ours-2B still loses 4.7% on MMStar (50.4 vs 52.9) and 9.7% on MathVista (50.3 vs 55.7), so "near-lossless" is benchmark-dependent rather than uniform.

• Transfer to other native backbones is mixed. SAIL-7B keeps a tolerable margin on general benchmarks but drops 13.6% on MMStar at 25%; ONECAT-3B, which folds spatial information into a single 1-D signal, drops 5.5% on AI2D versus 2.2% for SAIL-7B — the ordering is not uniform, motivating directional decoupling.

• PACE beats transferred modular methods, especially under aggressive compression. At 25% retention, FastV drops MMB from 69.7 to 50.4 (−27.7%); SparseVLM loses 6.5% on MathVista; PACE's advantage is largest at 25% where dimension-aware scoring matters most.

• Adaptive scheduling beats uniform allocation. At 25% overall retention the selected Pareto configuration (0.75, 0.67, 0.50) outperforms uniform compression by 3.6% on MMB and 7.6% on AI2D.

• Both spatial axes are needed. Removing spatial decoupling (using a single un-decoupled spatial score) costs 2%–4% on MMB; keeping only the width term causes a sharper decline at 25% retention.

• Three compression stages is the sweet spot. A single stage causes abrupt information loss (SEED drops 26.0% at 25% retention), two stages remain insufficient, and a fourth stage brings negligible gains with added scheduling complexity.

• Efficiency. At 25% retention (~256 tokens) measured on one H20-96G GPU at 1024² resolution, batch size 1: 2.74 T FLOPs versus 5.43 T for the full model (1.98×), KV cache reduced from 251.7 MB to 71.7 MB, GPU memory from 7.1 GB to 6.2 GB, while the average score rises from 65.4 to 65.6. Batch-1 decode speed is unchanged (1.00×) because it is weight-loading dominated. Under batched decode at 25% retention, PACE reaches up to 1.9× faster decoding and 2.0× lower peak memory and supports batch 64 where the full model runs out of memory (reported in Appendix F.5).

• Search space is tractable thanks to Finding 1. Three stages chosen from 36 post-IFB layers gives 7,140 layer configurations; a 25-value retention grid gives 15,625 ratio combinations and roughly 1.12×10⁸ total candidates; the monotonicity relation r₁ > r₂ > r₃ narrows traversal substantially.

Methodology in Plain English

The authors start by building their own native MLLM so they have a testbed that exposes every training stage: a 1.7B Qwen3-based backbone scaled to 2B parameters, trained in four stages (pre-training on ~518M samples with the Post-LLM backbone frozen, mid-training on ~89M samples, supervised fine-tuning on ~12.8M samples with all parameters optimized, and reinforcement learning on ~12K samples using GRPO with rule-based reward and category-proportional priority sampling for hard examples), on 5×8 H20-96G GPUs in bfloat16.

They then probe where redundancy lives. Rather than assuming visual tokens are uniformly redundant, they measure how much independent structure the token matrix spans at each layer, and crucially they measure it separately along image height and width. That tells them two things: the first few layers (the "Initial Fusion Buffer", measured at L_buf = 4) are busy aligning vision and language and should not be pruned hard, and a token can be redundant along one spatial axis while being essential along the other.

The compression itself is dead simple at inference: at three chosen layers after the IFB, keep the top-K visual tokens by an importance score. What makes the score unusual is that it is read straight out of the model's existing attention computation. Because Native-RoPE encodes time, height and width in separate slices of each query and key vector (with base frequencies β_T = 1×10⁶ and β_H = β_W = 1×10⁴), the attention score is literally a sum of a temporal, a height and a width dot product. The authors average each of these three terms over text queries and heads, apply ReLU, then combine them as (height + width) × temporal: adding the spatial terms means a token stays if it is salient in either direction, while multiplying by the temporal term suppresses tokens the text does not care about at all.

Choosing how hard to compress at each of the three stages is framed as a small multi-objective search: minimize performance loss on a calibration set, minimize spatial coverage variance across a 4×4 grid of image regions, and maximize total token reduction, subject to an overall retention target. Retention ratios are enumerated on a discrete grid, points are filtered for Pareto dominance, and one operating point is selected. Because the third stage sits deeper on the high-discriminability plateau, the search naturally returns decreasing retention with depth.

Finally, they test the whole recipe on eight benchmarks via VLMEvalKit (MMBench-DEV-EN, AI2D, MME, HallusionBench, RealWorldQA, MMStar, SEEDBench-IMG, MathVista), comparing against both native backbones (SAIL-7B, ONECAT-3B) and ViT-based ones (Qwen3-VL, InternVL3), and separately against modular compression methods transferred onto their own backbone.

Why This Matters

Impact on research. The paper redirects visual token compression from the well-trodden ViT-based setting to native MLLMs, which the pilot study shows have the most redundancy to give away. It also demonstrates that architectural priors — here, Native-RoPE's axis decomposition — can be repurposed as a free, parameter-free importance signal, which is a reusable design pattern rather than a one-off trick. The negative results matter too: transfer to ONECAT-3B (single 1-D spatial signal) is meaningfully worse, which is evidence that the method depends on genuine multi-axis encoding rather than being universally applicable.

Real-world applications:

• Long-document and chart understanding on-device, where high-resolution images produce thousands of visual tokens and KV-cache memory dominates cost.
• Batched multimodal serving, where the paper reports up to 1.9× faster decoding and 2.0× lower peak memory at 25% retention, and batch 64 becomes feasible where the full model runs out of memory.
• Edge and mobile assistants that must fit a native multimodal model in a fixed memory budget without a separate vision encoder.
• Video or multi-image inputs, where per-frame token counts multiply and the same compression schedule can be applied per frame.

Industry relevance. The work comes from ByteDance, and the framing is explicitly deployment-oriented: FLOPs, prefill latency, KV-cache megabytes and GPU memory at 1024² resolution on H20-96G hardware, not just benchmark deltas. The "accuracy at matched serving cost" argument — PACE scoring 65.6 versus SparseVLM's 64.6 and PyramidDrop's 62.7 at the same 256-token budget — is the kind of claim that matters to a serving team, and the fact that the method needs no retraining and adds no parameters makes it cheap to adopt.

Future Directions

• Handling spatially collapsed position encodings. ONECAT-3B's larger degradation (5.5% on AI2D at 25% retention versus 2.2% for SAIL-7B) suggests SDIE needs a fallback importance estimator for native backbones that fold spatial information into a single 1-D signal; a general recipe is not given.
• Recovering the math and dense-reasoning gap. MathVista remains the weak point across backbones (−9.7% for Ours-2B at 25%, −11.2% and −18.3% for SAIL-7B and ONECAT-3B), and MMStar also degrades. Whether a finer-grained importance signal or a task-conditional retention schedule can close this is open.
• Coupling compression with training rather than applying it only at inference. The paper trains a full PT/MT/SFT/RL pipeline around the compressed backbone but applies compression directly at inference; making the schedule part of RL or SFT optimization is a natural extension the four-stage setup now makes feasible.
• Scaling the analysis beyond a 2B model and 1.12×10⁸-candidate discrete search. Whether the depth-progressive ramp and the Pareto-selected schedule transfer to much larger native backbones, and whether the search can be done analytically instead of by grid traversal, are both unaddressed.

Target Audience

Researchers and engineers working on efficient multimodal inference — particularly those building or serving ViT-free / early-fusion multimodal models and those who own the latency and memory budget for vision tokens. It is also useful for compression researchers who want a clear demonstration that methods designed for ViT-based pipelines do not transfer cleanly, and for practitioners who need a training-free, parameter-free pruning scheme they can drop into an existing native backbone. Readers without background in RoPE and attention decomposition will find the method sections demanding, though the probing findings and the benchmark tables are accessible on their own.

Authors’ abstract

Native multimodal large language models (native MLLMs) jointly encode visual and linguistic sequences within a unified autoregressive Transformer, emerging as a promising paradigm for multimodal understanding. Unlike conventional ViT-based MLLMs relying on external vision encoders, ViT-free native MLLMs tokenize pixel patches directly, potentially suffering severe visual token redundancy. A pilot study confirms this: unlike ViT-based models that degrade under compression, ViT-free MLLMs tolerate aggressive compression with near-lossless or even improved performance. Yet existing compression methods target ViT-based settings, leaving native MLLMs, which stand to gain the most, unexplored. To close this gap, we introduce an analysis pipeline centered on token discriminability and cross-dimensional heterogeneous redundancy, revealing that (i) redundancy is distributed independently and heterogeneously along height and width axes, and (ii) early-layer visual tokens are indiscriminable and uniformly redundant. Guided by these findings, we propose PACE (Pareto-Adaptive Compression for Efficient Native MLLMs), the first visual token compression framework tailored for native MLLMs with Multimodal-RoPE. Leveraging dimensional decoupling of positional encoding, PACE decomposes attention into two orthogonal spatial saliency estimates along height and width, gated by a positional attention signal, yielding direction-aware importance with negligible overhead. It further casts token budget allocation as Pareto optimization over performance preservation, inference efficiency, and spatial coverage, jointly identifying optimal compression depths and retention ratios. Experiments show PACE maintains near-lossless performance at only 25%-50% visual token retention, even improving several benchmarks, while delivering over 2x FLOPs reduction.