Lost in Compression: A Controlled Cross-Lingual Audit of Extractive Prompt Compressors

AI summary

Overview

Research area: Natural Language Processing — specifically extractive prompt compression and multilingual (cross-lingual) evaluation of LLM context-reduction methods.

Technical level: Intermediate. The core idea is accessible without deep math, but readers benefit from familiarity with tokenizers, keep-rate budgets, and LLM evaluation harnesses.

Scope: A controlled, fully parallel ten-language audit of four learned prompt compressors against four deterministic baselines, across eleven target models from ten vendors, asking whether English-trained compression closes or widens the pre-existing "token premium" that non-English text already pays.

What This Paper Is About

Long prompts dominate the cost of deploying LLM applications, and extractive prompt compression — where a small model decides which tokens or sentences to keep before the prompt reaches the target model — promises to cut that cost. These compressors are developed, trained, and almost exclusively evaluated in English, even though subword tokenizers already make the same content cost 1.3–1.8× more o200k tokens in the nine non-English languages studied here. The paper asks a simple but unasked question: when an English-supervised compressor meets non-English text, does it close that cost gap or widen it?

Key Contributions

1. A controlled audit protocol that separates compressor quality from task difficulty, tokenizer effects, and target-model behavior, using paired parallel items, achieved-budget matching, and full-context / no-context / shuffled-context anchors.
2. Evidence that the transfer gap is rate-dependent and tied to compression supervision language, not architecture: significant in 8 of 9 non-English languages at a 0.33 keep-rate on both primary target models, replicating across nine further target models and across three English-supervised compressor backbones (XLM-R, mBERT, ModernBERT); deterministic methods show no comparable gap, and the multilingually trained XProvence v1 shows none at all.
3. A long-context stress test in which learned compression reduces non-English contexts to or below no-context performance.
4. A translation-arbitrage analysis showing that translate-then-compress can dominate native-language compression under stated caveats about translationese.

Main Findings

• The gap is real at deep budgets: At a 0.33 keep-rate, English retains 57–62% of normalized context utilization while Lithuanian retains 10–24% and Chinese essentially none (U = −0.03 on GPT-5.4-mini, meaning compressed context is worse than no context at all). The gap is significant in 8 of 9 non-English languages for both primary target models.
• The gap is rate-dependent: At a mild 0.75 keep-rate, languages behave nearly alike for GPT (only Chinese shows a significant gap). The safe compression budget (deepest rate with U ≥ 0.8) is about 2× compression for English but only roughly 1.3× or less for the other nine languages.
• Tokenizer pricing and compression safety are decoupled: Chinese has the smallest token premium in the set (1.28× under o200k) yet the largest compression penalty.
• Deterministic baselines do not show the gap: On GPT-5.4-mini no deterministic method (TF-IDF, truncation, random pruning, lemmatization+stopword removal) reaches significance in any of the nine languages at any rate (means −0.20 to −0.09 at keep-rate 0.5), versus +0.27 (7/9 significant) for LLMLingua-2. On Claude the deterministic means at 0.5 are −0.04 to +0.13 (at most 2/9 significant) versus +0.26 (9/9). At the deepest budget on Claude, truncation and random deletion do develop gaps (5/9 and 6/9, means +0.16/+0.19), but Chinese — the worst language for every learned method — is among the least affected (+0.04 to +0.06).
• Method rankings invert outside English: In English at 0.33, LLMLingua-2 is 12–14 accuracy points ahead of TF-IDF; across the nine non-English languages that advantage shrinks to between −5 and +7 points, and the ranking reverses in ET/LT/ZH on GPT and FI/ET/LT/ZH on Claude.
• Supervision language, not architecture, drives the gap: All three English-supervised compressors show it (mean gap +0.23 to +0.31 at keep-rate 0.5 on the primary model, up to 9 of 9 languages significant). Kompress-v2, the ModernBERT prose compressor inside the Headroom production stack, shows a significant gap in 8/8 auditable non-English languages on both primary models (mean +0.29 GPT, +0.39 Claude). XProvence v1, trained multilingually, shows no gap on GPT-5.4-mini at either threshold (mean −0.03 at threshold 0.5, no language significant, n=300), replicating on Claude at threshold 0.1 (mean +0.005, 0/9 significant).
• The fix is not stable across releases: XProvence v2, retrained on translated rather than natively multilingual data, empties 92% of Chinese contexts at its aggressive threshold (achieved rate 0.08, versus 0.21–0.24 in other languages) with no error or warning, yet keeps essentially the whole passage (achieved rate 1.02) at its conservative threshold. Its Chinese gap reaches up to +0.97 on Claude.
• Budget control itself fails cross-lingually: The mBERT variant requesting 0.75 on Chinese achieves a rate of 1.20 — the "compressed" text is longer than the original. The XLM-R variant fails the other way, over-compressing (achieved 0.247 at requested 0.33, versus 0.307 for English). Kompress-v2 is effectively a no-op on Chinese (achieved 0.91–0.96) because its whitespace tokenization cannot segment Han script; LLMLingua-1 was excluded entirely after achieving rate 0.99 on Lithuanian.
• Replication across eleven target models: Every model shows the same qualitative ordering (EN safest, ZH consistently hardest), with mean gaps at keep-rate 0.5 of +0.26 to +0.40 (per-language gaps up to +0.82). Newer models do not help: GPT-5.6-luna handles compressed non-English input no better than the older mini model (mean gap +0.35 vs +0.27). The four Chinese-vendor models suffer the largest Chinese gaps of all (DeepSeek +0.82, Kimi +0.72, MiniMax +0.70, Qwen +0.72), so the failure sits in the compressor rather than in the target model's command of Chinese.
• Refusals are a compression-induced failure mode: Claude's refusals are ≤0.7% in every language with the full passage, but rise to up to 11% of truncated/randomly-pruned Chinese calls, and 9–31% of no-context calls (27.7% English, 31.3% Chinese). GPT-5.4-mini answers regardless.
• Long context makes it worse: With GPT, compressed Lithuanian, Latvian, and Polish contexts perform at or below the no-context anchor (U ≤ 0). Claude replicates the ordering (EN 0.56 vs 0.14–0.33 elsewhere). Deterministic lemma+stopword compression preserves 44–66% everywhere.
• It is task-dependent: On MultiEURLEX topic classification, compression is nearly free in all languages down to 20×, and a title-only heuristic at a 5% budget matches or beats the full document (micro-F1 0.47–0.51 vs 0.45–0.48).
• Translation arbitrage works: Translation alone saves 35–44% of tokens at a 5–8 pp quality cost, and translate-then-compress delivers, at roughly 0.18× the native token cost, accuracy that matches or beats native-language compression at 0.33× cost in three of five languages (LT +10 pp, FI +6.6, ET +5).
• Mechanism: A retention diagnostic suggests the compressor disrupts surface carriers of grammatical structure rather than simply deleting facts. At 0.33, number-bearing tokens are retained more often in Lithuanian (0.70), Latvian (0.78), and Polish (0.70) than English (0.61), yet quality falls far more; capitalized-word retention (a named-entity proxy) is highest in English (0.86 vs 0.72–0.79).
• Input length is ruled out: The compressor's internal XLM-R tokenizer prices all ten languages nearly equally (0.97–1.25), so the compressor receives balanced input lengths.

Methodology in Plain English

The study fixes the semantic content and varies only the language. Ten fully parallel languages (EN, PL, FI, ET, LV, LT, UK, ZH, AR, HI) cover four language families, seven branches, and five scripts, with o200k token premiums from 1.28 (ZH) to 1.83 (LV), and up to 4.5× under Qwen2.5's tokenizer (Hindi). The primary task is Belebele reading comprehension with 300 parallel items per language; auxiliary arms add MultiEURLEX level-1 EUROVOC classification (24 parallel documents) and a long-context task (a target passage among 7 same-language distractors, roughly 2–3k tokens, 150 items), both over six core European languages.

Every item is run under: full context, no context (a contamination/prior anchor), LLMLingua-2 at requested keep-rates of 0.75, 0.5, and 0.33, TF-IDF sentence extraction at the same rates, deterministic lemmatization+stopword removal (its natural budget is measured, not controlled), plus two controls matched to LLMLingua-2's achieved o200k budget per item: prefix truncation and seeded random word deletion. Three further learned systems are audited: Kompress-v2 and both XProvence checkpoints.

The key metric normalizes accuracy against both anchors, giving the fraction of usable context value retained, U. The "transfer gap" is English's U minus the language's U. Uncertainty is estimated with item-level paired bootstrap (2,000 resamples) on identical items across languages, with 95% CIs reported per comparison and no multiplicity correction — which is why the authors emphasize patterns that replicate across models, methods, and rates rather than any single interval.

Compression is a cost technique, so the audit targets the cost-efficient deployment tier (mini/flash/lite class). Primary targets are gpt-5.4-mini and claude-haiku-4-5 on the full grid (n=300). Nine further targets run a reduced grid (n=150; full, no_context, LLMLingua-2, TF-IDF, truncation at 0.5 and 0.33): gpt-5.6-luna, gemini-3.5-flash, Llama 4 Maverick, Mistral Medium 3.5, DeepSeek V4 Flash, Kimi K2.6, MiniMax M3, Nova 2 Lite, and Qwen 3.7 Plus. Temperature is 0; instructions are held constant in English while passages, questions, and options are in the item language.

Why This Matters

Impact on research. This is described as the first controlled cross-lingual audit of extractive prompt compression. It shows that the cross-lingual axis — unlike transfer to diffusion LLM targets, which has been audited — was untested, and that an English-only evaluation would rank compressor methods in the wrong order for a large share of the ten languages. It also reframes an existing deployment-side observation (a Chinese community fork of Headroom adding a Chinese lane; upstream merging CJK-aware segmentation into its non-learned text lane while the learned Kompress-v2 lane remains whitespace-based as of v0.32) as a quantified, reproducible failure mode.

Real-world applications:

• RAG and document QA in non-English languages: aggressive learned compression can push contexts to or below no-context utility while the user still pays for the tokens.
• Production cost routing: teams should report and monitor achieved-versus-requested keep-rate per language, since no-ops, over-compression, empty outputs, and budget drift all occur with no error or warning.
• Chinese-language services: whitespace-based or English-supervised compressors should never be applied — they are silent no-ops or produce empty contexts.
• Translate-then-compress pipelines: for batch or cache-warm workloads, translating to English before compressing (or using a native multilingual pruner) can dominate native-language compression; note both off-the-shelf remedies tested (XProvence, CC BY-NC-ND 4.0; NLLB-200, CC BY-NC 4.0) are non-commercial.
• Task routing: for surface-signal tasks such as topic classification, positional heuristics at a 5% budget match the full document, so aggressive compression is safe regardless of language.

Industry relevance. The audit covers the deployment tier where most real traffic by production token volume runs, and includes a production system (Headroom's Kompress-v2) alongside proprietary and open-weights models from ten vendors. It documents a failure that produces no error signal, plus economics the audit flags: compressing a shared, prefix-cached context saves less than naive token arithmetic suggests, and query-aware pruning forfeits prefix-cache reuse entirely.

Future Directions

• A native-source replication of the translation-arbitrage arm. The current passages descend from FLORES (translated from English), so round-trip translationese may flatter the translate-then-compress pipeline; NLLB-200 was itself developed around FLORES-200, so its quality here is likely an upper bound.
• A case-marking minimal-pair probe to test the proposed mechanism directly for Baltic, Slavic, and Finnic languages; the authors report it is under native-speaker validation. A separate Chinese diagnostic is future work, since Chinese requires a different account (multi-character words, grammatical particles as high-frequency "function tokens").
• Stabilizing multilingual supervision. XProvence v1 closes the gap but v2, retrained on translated data, does not at aggressive thresholds, showing the fix is not stable. The paper frames the open question as calibrating relevance scores per language, and notes fine-grained Chinese pruning is not achievable with XProvence as shipped (its sentence splitter treats each Chinese passage as a single unit, mean 1.0 segments vs 3.4–4.3 in other scripts).
• Broadening language and context coverage. The study covers ten languages and five scripts, with no non-Uralic agglutinative language such as Turkish; the auxiliary arms cover only six core European languages; contexts here are ≤3k tokens while production RAG contexts are often 10–100× longer; and instructions were held constant in English rather than localized.

Target Audience

Practitioners deploying RAG, long-context, or cost-optimized LLM pipelines outside English, who need concrete budget guidance and monitoring advice; NLP researchers working on prompt compression, tokenization fairness, or multilingual evaluation; and evaluation engineers who design benchmarks and want to see why single-rate English scores can mislead. Readers who want architecture-level novelty will find little — the contribution is a controlled audit and a supervision-data attribution, not a new compressor.

Authors’ abstract

Extractive prompt compression promises to cut LLM inference costs by removing low-information tokens, and learned compressors such as LLMLingua-2 report strong results on English benchmarks. Most other languages already pay a token premium: the same content costs 1.3-1.8x more tokens than in English. We ask whether compression closes or widens this gap. Using fully parallel data in ten languages spanning five scripts, with controls budget-matched in the target model's tokenizer, we audit four learned compressors against four deterministic baselines, on eleven target models from ten vendors (over 250,000 evaluation calls). Three of the compressors are trained with English supervision (LLMLingua-2 XLM-R/mBERT; Kompress-v2 from the production Headroom stack); the fourth, XProvence, is trained multilingually. First, the transfer gap is real, replicates across target models and compressor backbones, and is strongly rate-dependent: at a 0.33 keep-rate English retains 57-62% of normalized context utilization while Lithuanian retains 10-24% and Chinese essentially none, despite Chinese having the smallest token premium. Second, the gap tracks compression supervision data, not architecture. All three English-trained compressors show it, deterministic methods show no comparable gap, and the multilingually trained XProvence v1 shows none. Its v2 release, retrained on translated data, empties 92% of Chinese contexts at its aggressive threshold without any warning. Third, in a harder long-context setting, aggressive learned compression drives compressed contexts to or below no-context utility in three of five non-English languages. A translate-then-compress pipeline matches or beats native compression at roughly half the token cost in three of five tested languages. We release all code, compressions, and model outputs. Safe compression budgets are much smaller outside English.