PPE: Positional Preservation Embedding for Token Compression in Multimodal Large Language Models

AI summary

Overview

• Research area: Efficiency of multimodal large language models (MLLMs) — specifically visual token compression and positional encoding for vision-language reasoning.
• Technical level: Intermediate. It assumes familiarity with Transformer attention, Rotary Position Embeddings (RoPE), and standard vision-language benchmarks, but the core idea (splitting positional IDs across dimensions) is explained concretely.
• Scope: The paper introduces Positional Preservation Embedding (PPE), a parameter-free operator that lets a single compressed visual token carry multiple original spatial/temporal position IDs so that token merging no longer destroys image layouts or video temporal order.

What This Paper Is About

Multimodal large language models encode images and video frames as dense sequences of visual tokens, which are highly redundant and therefore slow and memory-heavy. A common fix is to merge similar tokens, but merging usually throws away the positional relationships of the merged tokens, breaking layout-sensitive tasks such as reading text in an image or reasoning over a video's temporal order. This paper proposes a way to compress tokens while keeping multiple original positions encoded inside each surviving token.

Key Contributions

1. Problem identification: The authors point out that existing visual token merging methods neglect spatial structure preservation and temporal coherence, which distorts intra-frame layouts and disrupts inter-frame temporal relations.
2. PPE operator: A novel, plug-and-play, parameter-free positional encoding that explicitly preserves spatiotemporal integrity during token merging, allowing one compressed token to represent several original position IDs rather than one.
3. Cascade compression support: PPE is decoupled from the merging algorithm and can be applied progressively across multiple Transformer layers of an MLLM, enabling much higher compression with minimal performance loss.
4. Broad empirical validation: Experiments across image and video benchmarks show PPE reducing visual token count by 90% while outperforming other compression methods at comparable reduction ratios, with consistent gains of 2%~5%.

Main Findings

• Image benchmarks at 55% token reduction: Integrating PPE into the Chat-UniVi clustering framework raises the image average from 67.61 to 79.32 (+11.71), against a dense baseline of 82.96. The biggest jumps are on layout/OCR-heavy tasks: TextVQA 57.66 to 77.14 (+19.48), DocVQA 52.48 to 76.79 (+24.31), ChartQA 49.60 to 74.52 (+24.92), and OCRBench 535 to 691 (+156). MMBench (CN) improves from 83.71 to 84.87 (+1.16), while MMBench (EN) is essentially flat at 84.73 (-0.19) versus 84.92.
• Video benchmarks: At 55% reduction with spatial-only compression, PPE raises the video average from 56.74 to 58.69 (+1.95) and even edges past the dense baseline of 58.52. At 94% reduction with spatiotemporal compression, PPE reaches a video average of 57.62 versus 55.70 for Chat-UniVi (+1.92). Notable gains include NeXT-QA (OE) 25.37 to 32.61 (+7.24) at 55%, and MVBench 64.38 to 66.42 (+2.04) at 94%.
• PPE versus M-RoPE: Since the dense model uses M-RoPE by default, Chat-UniVi inherits it; PPE instead retains positions of merged tokens within a single representation and is reported to be consistently stronger at identical reduction ratios.
• Cascade (within-LLM) compression: In Qwen2.5-VL-3B-Instruct, which has 36 transformer layers, inserting PPE clustering modules at layers 11, 23, and 35 with a per-layer clustering ratio of 0.45 achieves 90% token reduction while keeping performance comparable to the 55% case. At a 90% budget, the within-LLM configuration averages 58.48 versus 57.69 for the other 90% configuration reported, showing cascade compression beats compressing only once before the LLM.
• Plug-and-play compatibility: On Qwen2-VL-7B-Instruct, adding PPE to PACT improves MMBench (EN) 74.14 to 74.48, MMBench (CN) 74.17 to 75.00, and TextVQA 73.73 to 73.87 at an 89% reduction ratio; adding it to ToMe improves MMBench (EN) 74.31 to 74.57, MMBench (CN) 73.63 to 74.74, and TextVQA 74.94 to 76.16 at a 57% reduction ratio. Generation time changes only marginally (PACT 0.08s to 0.09s; ToMe 0.90s to 0.91s) and peak memory is unchanged (15.82 GB for PACT, 15.81 GB for ToMe).
• VisionZip and training-free vs. SFT results (Qwen2.5-VL-3B-Instruct, 55% reduction): In the training-free setting, Chat-UniVi+PPE gains +36.18 on TextVQA, +46.58 on DocVQA, +48.52 on ChartQA, and +291 on OCRBench over Chat-UniVi. With SFT, Chat-UniVi+PPE gains +2.04 on TextVQA, +5.68 on DocVQA, and +2.16 on ChartQA over VisionZip, and +69 on OCRBench.
• Model-size generalization (Qwen2.5-VL-7B-Instruct): PPE helps at 7B as well. In the SFT setting, Chat-UniVi+PPE improves TextVQA from 54.92 to 82.46 (+27.54), DocVQA from 50.01 to 85.84 (+35.83), and ChartQA from 43.96 to 78.88 (+34.92). One reported exception is VisionZip+PPE on ChartQA in the training-free 7B setting, which drops slightly (60.56 to 59.88, -0.68), a drop the authors attribute to perturbed attention without SFT; after SFT it becomes an improvement.
• Backbone generalization: On LLaVA-OV-0.5B evaluated on VideoMME (w/o subs), PPE reaches 44.74 average at 96% reduction versus 44.80 for Dense and 43.07 for Chat-UniVi, with gains on the Short (56.40 vs. 53.00) and Medium (42.10 vs. 40.40) splits.
• The capacity hyperparameter K matters: With K=1 the average is 57.97 and only 45% of IDs are retained; K=8 gives the best average of 58.69 with 77% ID retention; K=24 gives 58.28 with 84% ID retention. K=8 matches the greatest common divisor of the 3D M-RoPE sections [16, 24, 24] and is the default.
• Why it works (attention analysis): On TextVQA samples at 55% compression, both Chat-UniVi and PPE reduce attention entropy relative to the dense baseline, but PPE shows consistently higher variance, indicating sharper grounding. In case 34602 ("What is the brand of this camera?"), Chat-UniVi's attention is narrowly confined and answers incorrectly with "d", while PPE covers the text and recovers "dakota digital".
• Reduction-ratio and aggregation ablation: Averages across video benchmarks are 58.22 at 25%, 57.99 at 40%, 58.69 at 55%, 58.26 at 70%, and 57.69

Authors’ abstract

Multimodal large language models (MLLMs) have achieved strong performance on vision-language tasks, yet often suffer from inefficiencies due to redundant visual tokens. Existing token merging methods reduce sequence length but frequently disrupt spatial layouts and temporal continuity by disregarding positional relationships. In this work, we propose a novel encoding operator dubbed as \textbf{P}ositional \textbf{P}reservation \textbf{E}mbedding (\textbf{PPE}), which has the main hallmark of preservation of spatiotemporal structure during visual token compression. PPE explicitly introduces the disentangled encoding of 3D positions in the token dimension, enabling each compressed token to encapsulate different positions from multiple original tokens. Furthermore, we show that PPE can effectively support cascade clustering -- a progressive token compression strategy that leads to better performance retention. PPE is a parameter-free and generic operator that can be seamlessly integrated into existing token merging methods without any adjustments. Applied to state-of-the-art token merging framework, PPE achieves consistent improvements of $2\%\sim5\%$ across multiple vision-language benchmarks, including MMBench (general vision understanding), TextVQA (layout understanding) and VideoMME (temporal understanding). These results demonstrate that preserving positional cues is critical for efficient and effective MLLM reasoning. Our code is available at https://github.com/MouxiaoHuang/PPE.