Proxy Compression for Language Modeling

AI summary

Overview

Research area: Natural Language Processing — language model training objectives, input representations, byte-level modeling, and tokenization.

Technical level: Advanced. The paper assumes familiarity with tokenizers (BPE, WordPiece, UnigramLM), arithmetic coding, byte-level language models, and compute-matched training comparisons.

Scope: The paper introduces "proxy compression," a mixed-representation training scheme that trains a single language model jointly on raw UTF-8 bytes and externally compressed sequences (90% of samples compressed by default), then discards the compressor and runs inference on raw bytes alone.

What This Paper Is About

Almost all modern language models are trained on sequences produced by a fixed external compressor — usually a tokenizer — which hard-wires that compressor into every input and output. This makes the model efficient to train but not truly end-to-end at the raw-data level, and it inherits known tokenizer artifacts such as prompt boundary problems, glitch tokens, and biases against low-resource languages.

The paper asks whether a model can keep the training-efficiency benefits of compressed inputs while still presenting a plain raw-byte interface at inference time. Its answer is to treat the external compressor as a training-time proxy only: the model is trained on both compressed and raw views and learns to align them internally.

Key Contributions

1. Proxy compression as a training scheme. A mixed-representation training method that trains one model with next-symbol prediction over both compressed sequences and raw UTF-8 bytes, requiring no architectural changes — all modifications live in the data preprocessing pipeline. Raw and compressed sequences are wrapped in sentinel tokens (⟨raw⟩, ⟨/raw⟩, ⟨comp⟩, ⟨/comp⟩) so the model can condition on representation type.

2. Demonstration of strong cross-representation transfer. Models that see only 10% of training samples as raw bytes nonetheless outperform pure byte-level baselines under fixed compute budgets at sizes above 1.5B parameters, and at larger scales match or surpass tokenizer-based baselines while operating solely on raw bytes.

3. A systematic comparison of proxy compressor families. Tokenizer-based proxies (OpenCoder tokenizer, ~2.9x compression), neural proxies (arithmetic coding on a ~40M-parameter byte-level model, ~2.6x compression), and generic gzip proxies (~2.5x compression). Tokenizer-based and neural proxies transfer strongly; gzip fails to transfer and even degrades performance.

4. Analysis of why transfer works. In-context translation probing, compressor stability measurement under perturbation, collision statistics for neural compression, robustness evaluation on the ReCode benchmark, and mixing-ratio ablations that reveal asymmetric transfer direction.

Main Findings

• Transfer scales with model size. On HumanEval-Plus with a fixed budget of 100B symbols, the byte-level baseline scores 15.9 (0.5B), 18.3 (1.5B), 22.0 (4B), 23.8 (7B), 24.4 (14B), while Proxy (Tokenizer) scores 12.2, 20.7, 24.4, 26.2, 30.5 and Proxy (Neural) scores 13.4, 18.3, 22.6, 26.8, 29.9. The tokenizer-based reference scores 17.7, 18.3, 28.0, 28.7, 29.3. At small scale (0.5B) proxy models underperform byte-level; above 1.5B they overtake it.

• MBPP-Plus shows the same pattern. Byte-level: 25.9, 33.6, 41.8, 41.3, 42.1 across 0.5B–14B. Proxy (Tokenizer): 25.4, 38.4, 44.4, 45.5, 49.5. Proxy (Neural): 22.0, 29.6, 41.8, 41.8, 49.2. The tokenizer baseline: 29.4, 41.0, 46.3, 45.2, 48.1 — so at 14B both proxies exceed it.

• Longer training horizons preserve the trend. After 320B symbols on the full RefineCode GitHub split, at 1.5B the proxy models (Neural 14.0, Tokenizer 12.8 on HumanEval-Plus) beat the byte baseline (9.1) but trail the tokenizer model (17.1). At 7B, proxy models reach 21.3 (Neural) and 22.0 (Tokenizer), matching or exceeding the tokenizer baseline (21.3) while the byte baseline scores 14.6.

• Transfer can be in-context, not only in weights. With no paired examples, oracle-translation pass@1 at 50k steps reaches ~46% (tokenizer) and ~33% (neural). With pairing always on, both compressors reach above 95% pass@1. Under warm-up-only pairing, translation accuracy decays toward the no-pairs level but ordinary pass@1 is slightly better than always-on (20.7 vs. 17.0 for tokenizer; 18.9 vs. 14.6 for neural), suggesting high in-context translation accuracy is not necessary for strong downstream results.

• Compressor stability predicts transfer quality. Under random 10% character deletion on 80K samples, measured by normalized Levenshtein distance between compressed outputs, tokenization is highly stable, gzip is significantly more unstable, and neural compression lies in between.

• Gzip proxies fail. For 1.5B models, increasing the proportion of gzip-compressed data degrades performance — models trained on 0% gzip always outperform gzip-mixed variants. The authors attribute this to unstable, unstructured outputs that resemble noise to a language model.

• Neural compression is "fuzzily" structured. Neurally compressed sequences are not uniquely decodable: one compressed segment can correspond to multiple raw byte chunks (collisions). However, over 90% of collisions have longest-common-prefix ratios above 0.8, meaning colliding chunks are nearly identical except for short suffixes — ambiguity that groups formatting differences while preserving semantics.

• Robustness is largely retained. On the HumanEval split of ReCode with 7B models, macro-averaged robust pass@1 is 14.9 (tokenizer baseline), 18.7 (byte-level), 19.1 (Proxy Tokenizer), and 19.8 (Proxy Neural). Standard pass@1 is 32.9, 26.2, 32.9, and 30.5 respectively. Format perturbations (whitespace, indentation) drive the largest gap.

• Transfer is asymmetric. In the mixing-ratio ablation with 1.5B tokenizer-proxy models, byte-level inference remains strong even with only 10% raw data, while token-level inference degrades nearly monotonically as raw-byte proportion increases.

• Compression buys more unique documents. At r = 0.9, models observe roughly 3x more samples within a fixed compute budget than at r = 0.0 (100% raw bytes).

• Byte-level inference is competitive at inference time. For 14B tokenizer-proxy models, byte-level inference matches or outperforms token-level inference in many cases, despite only 10% of training samples being raw bytes — attributed to strong transfer plus more test-time compute per problem from longer byte sequences.

Methodology in Plain English

The researchers take a training corpus and, for each sample, randomly decide whether to present it in compressed form (with probability r) or as raw UTF-8 bytes. The default is r = 0.9, so 90% of samples are compressed. Both forms can appear inside the same packed context, and one standard autoregressive model is trained on all of it with ordinary next-symbol prediction — no changes to the architecture.

To make the format explicit, each sequence is wrapped in special marker tokens. During an initial warm-up of 10k steps, the scheme also occasionally places both views of the same sample next to each other in one context (with order randomized), which encourages the model to translate between formats in context; r is ramped linearly from 0.4 to 0.9 during this phase, after which pairing is turned off and r stays at 0.9. The vocabulary is partitioned as 64 sentinel slots, then 256 slots for UTF-8 byte values, then compressed symbols.

Three compressor families serve as the "proxy." Tokenizer-based proxies simply use the OpenCoder tokenizer's 96,640 symbols. Neural proxies train a small (~40M parameter) byte-level model, use it to estimate per-byte probabilities, and apply arithmetic coding with equal-information windows; to avoid the sequential bottleneck of arithmetic coding at scale, the authors segment inputs at high-entropy byte positions and compress segments in parallel. Neural output symbols pack 16 bits each into a 65,536-way alphabet. Gzip proxies just apply gzip.compress(..., mtime=0) for a byte-level alphabet.

At inference the compressor is thrown away entirely, and the model runs on raw bytes. Experiments use the RefineCode corpus — roughly 270 GB of Python for the main comparisons and about 3.3 TB of multi-language code from the full GitHub split for longer runs. All models follow the EvaByte architecture at 0.5B, 1.5B, 4B, 7B, and 14B parameters, trained for 50K steps with a batch of 2M sequence symbols, giving a comparable FLOPs budget across representations. Evaluation is on HumanEval, MBPP, and their EvalPlus variants. Baselines are a pure byte-level model and a BPE tokenizer-based model.

Why This Matters

Impact on research. The work reframes the tokenizer as a tunable training-time ingredient rather than a permanent component of the modeling stack. It also provides a new empirical handle on representation transfer — showing that transfer strength grows with model scale, that transfer can be asymmetric, and that "fuzzy," non-invertible representations can outperform lossless ones for training when the ambiguity is structured. It connects the byte-level modeling literature with the compression-as-prediction view of language models.

Real-world applications:

• Code assistants and IDE copilots that need to handle arbitrary raw text without tokenizer-induced prompt boundary artifacts.
• Systems serving many languages, including low-resource ones where hand-crafted tokenizers are known to introduce fairness and coverage problems.
• Robustness-sensitive deployments (e.g., code completion against reformatted, renamed, or paraphrased inputs), where the paper reports proxy models retaining byte-level robustness.
• General-purpose models operating directly on raw byte streams, which the paper notes extends to various digital file contents beyond text.

Industry relevance. Training efficiency is a direct cost lever: the reported ~3x increase in unique samples seen within a fixed compute budget at r = 0.9 is a substantial throughput argument. At the same time, the paper is candid that byte-level inference has its own deployment cost, and that efficiency depends on the underlying byte-level architecture — EvaByte is used specifically to mitigate this through efficient attention and multi-byte prediction.

Future Directions

• Broader domain validation. The paper's evaluation centers on code, with complementary natural-language experiments at 1.5B parameters; larger-scale natural-language, multilingual, and mixed-domain validation is explicitly left as future work.
• Characterizing the compression–transfer–compute trade-off. The authors state that the trade-offs among compression rate, transfer strength, and compute efficiency are not yet fully characterized, noting that more aggressive compression may amplify efficiency gains but could degrade transfer quality.
• Architectural integration. Incorporating proxy compression directly into model architecture design, rather than treating it purely as data preprocessing, may unlock further gains.
• Understanding why gzip fails and what defines an effective proxy. The stability analysis and collision statistics are a first step, but the general criteria distinguishing useful from useless proxy compressors remain an open question — as does why warm-up-only pairing outperforms always-on pairing on downstream pass@1.

Target Audience

Researchers and engineers working on language model training objectives, byte-level and tokenizer-free modeling, input representation design, and efficient training pipelines. It is most useful to readers already comfortable with tokenization schemes, arithmetic coding, and compute-matched scaling comparisons, and to practitioners who care about removing tokenizer coupling from deployed models without giving up compressed-input training efficiency.

Authors’ abstract

Modern language models are trained almost exclusively on token sequences produced by a fixed tokenizer, an external lossless compressor often over UTF-8 byte sequences, thereby coupling the model to that compressor. This work introduces proxy compression, an alternative training scheme that preserves the efficiency benefits of compressed inputs while providing an end-to-end, raw-byte interface at inference time. During training, a single language model is jointly trained on raw byte sequences and compressed views generated by external compressors; through the process, the model learns to internally align compressed sequences and raw bytes. This alignment enables strong transfer between the two formats, even when training predominantly on compressed inputs that are discarded at inference. Extensive experiments on code language modeling demonstrate that proxy compression substantially improves training efficiency and significantly outperforms pure byte-level baselines given fixed compute budgets. As model scale increases, these gains become more pronounced, and proxy-trained models eventually match or surpass tokenizer approaches, all while operating solely on raw bytes and retaining the inherent robustness of byte-level modeling. Our code is available at https://github.com/LZhengisme/proxy-compression.