Information Capacity: Evaluating the Efficiency of Large Language Models via Text Compression

AI summary

Overview

• Research area: Large language model evaluation and inference efficiency, grounded in lossless text compression and information theory.
• Technical level: Advanced. The paper defines a quantitative metric involving entropy coding, negative log-likelihood, and analytic inference FLOPs formulas, though the motivation and main conclusions are accessible to non-specialists.
• Scope: The paper proposes "information capacity," a compression-based efficiency metric that incorporates tokenizer efficiency, and evaluates it on 56 open-source models across 5 heterogeneous datasets.

What This Paper Is About

As LLMs grow and test-time scaling expands input and output lengths, the field lacks a rigorous way to compare inference efficiency across models with different tokenizers, parameter counts, and architectures. Existing metrics struggle to bridge parameter count and actual inference cost. The authors exploit the link between compression and intelligence: because decoder-only LLMs are next-token probability estimators, and entropy coding turns predicted probabilities into bitstreams, a model's compression performance relative to its computational complexity becomes a measure of efficiency.

Key Contributions

1. A new efficiency metric. The paper defines information capacity as model intelligence (data size saved via compression) divided by model inference complexity (inference FLOPs, measured on a logarithmic scale), with a per-dataset negative offset in the numerator so that models of different sizes within a series receive nearly identical scores.
2. Tokenizer efficiency as a first-class factor. Unlike prior metrics such as capability density, information capacity accounts for how efficiently a tokenizer represents text, since fewer tokens reduce input and output counts, cost, and latency.
3. Large-scale evaluation. The authors assess 56 open-source models across 5 heterogeneous datasets (Mixed text, FinePDFs-en, Ch-FineWeb-Edu, FineWeb-Edu, NextCoder), releasing code and data publicly.
4. Ablations of contributing factors. The paper isolates the influence of tokenizer efficiency, pretraining data volume, MoE architecture, post-training, test sample length, and softmax temperature.

Main Findings

• Consistent capacity within a series. A series of models with varying sizes exhibits a consistent information capacity, which supports comparing efficiency across model series and predicting the performance of a different-sized model within a series using measured results.
• MoE leads on multiple datasets. The latest mixture-of-experts models, exemplified by DeepSeek-V3.1 and GLM-4.5, achieve the highest information capacity on multiple datasets, followed by the latest dense models such as Qwen3, Hunyuan, and GLM-4. On Mixed text, GLM-4.5-Base ranks 1st (IC 0.2415), DeepSeek-V3.1-Base 2nd (0.2396), Hunyuan-Pretrain 3rd (0.2331), GLM-4-Base 4th (0.2267), and Qwen3-Base (Dense) 5th (0.2056).
• Strong linguistic and domain biases. Rankings shift substantially across datasets and languages. The Llama series (Meta) and Gemma series (Google) perform poorly on the Chinese Ch-FineWeb-Edu corpus compared to models from Chinese companies; rankings also change between the NextCoder code dataset and the English FinePDFs-en dataset even though the code uses English characters. FinePDFs-en (PDFs) and FineWeb-Edu (webpages) are both English, yet cause slight ranking variation.
• Tokenizer efficiency dominates. Information capacity scales almost linearly with average text size per token across multiple datasets, with Pearson correlation coefficients consistently exceeding 0.98. In the mixed dataset, average text size per token ranges from 32.35 bits (Gemma-3) to 34.94 bits (DeepSeek-V3.1), a range of 2.59 bits, while average NLL for 7B–8B models ranges only from 2.822 bits (Llama-3.1-8B) to 3.155 bits (InternLM2.5-7B), a difference of 0.333 bits.
• More pretraining data helps, with diminishing returns. For TinyLlama-1.1B, increasing pretraining tokens from 0.5T to 3T steadily lowered NLL and raised information capacity, but gains were uneven: the steps from 0.5T to 1T and from 2T to 2.5T produced significant improvements, while others produced slight gains, presumably due to quality variation across pretraining data portions. Qwen2.5 (about 18T tokens) showed only slight information capacity gains over Qwen2 (7T tokens) despite identical architecture, indicating diminishing returns on already sufficient high-quality data.
• MoE architecture raises capacity, and lower activated-parameter fractions help further. On Qwen1.5, Qwen1.5-MoE (14.3B total, 2.7B activated, 18.9%) reached NLL 2.895 with FLOPs of 4.850 and IC 0.2146, versus 0.2022 for the 1.8B dense model. On Qwen2, Qwen2-MoE (57B total, 14B activated, 24.6%) reached NLL 2.668, FLOPs 26.676, IC 0.2059. For Llama-4, both the 109B and 400B variants activate 17B parameters and have equal FLOPs, but the 400B variant achieved an average NLL of 3.907 versus 3.977, raising IC from 0.1836 to 0.1859 (Table 4; the body text states 0.1856).
• Attention-variant choice matters little at this scale. For example, the MLA used by DeepSeek models reduces attention complexity for long sequences relative to GQA, but at the relatively small evaluation sequence length, attention is only a small portion of inference FLOPs, and complexity is measured logarithmically, so attention implementation has marginal influence on information capacity.
• Post-training degrades compression. Post-training impairs next-token prediction on plain text and lowers information capacity. The Qwen3 series uses multi-stage reinforcement learning, causing more severe degradation. Seed-OSS-36B-Base-woSyn shows a marginal drop from 0.17633 to 0.17628 after incorporating synthetic instruction data, while the Instruct variant measures 0.15995, well below the Base variant.
• Longer test samples marginally improve capacity. Raising text length from 1024 to 2048 lowered NLL and slightly increased per-token FLOPs, giving small IC gains (from 0.0003 to 0.0013 across Qwen3, Llama-3, GLM-4, and Seed-OSS models). The overall difference due to test sample length is negligible compared to differences between models.
• Baseline models are not always evaluable. Several influential base models are not publicly released, including gpt-oss from OpenAI, Phi-4 from Microsoft, and MoE variants of the Qwen3 series, so their information capacity could not be measured accurately.
• Benchmark correlation. The paper states that empirical results show a correlation between information capacity and benchmark scores, though the specific benchmark figures are not contained in the supplied content.

Methodology in Plain English

The authors treat an LLM as a probability estimator feeding an entropy coder. For each text sample, the model assigns a probability to each actual next token; the closer the predicted probabilities are to the truth, the shorter the encoded bitstream can be, following Shannon's source coding theorem. They measure the original text size using UTF-8 byte counts, then subtract the summed negative log-likelihood of the tokens the model actually saw to get a "compression gain." They divide that gain by inference FLOPs computed analytically from architecture hyperparameters (hidden size, key-value dimension, FFN dimension, logits dimension, number of blocks), with special formulas for Llama-4's auxiliary FFNs and DeepSeek's multi-head latent attention.

Because raw information capacity falls almost linearly as FLOPs grow, the authors normalize by token count, drop the first token (since no preceding context exists), and add a fixed per-dataset negative offset. This offset makes same-series models score nearly identically without changing model-series rankings. Measurements truncate samples to 1024 tokens, use the default temperature of 1, upcast logits from bfloat16 to float32 for numerical precision, and truncate ineffective logits beyond the tokenizer vocabulary before softmax. Datasets were filtered so all samples exceed the truncation threshold for every evaluated model.

Why This Matters

• Research impact: Provides a single metric that spans model sizes and architectures (dense and MoE), allowing cross-scale comparison where parameter count and tokenizer design previously blocked fair inference-cost comparison. It also reframes tokenizer design as an efficiency lever, not an implementation detail.
• Real-world applications:
  • Choosing which model to deploy when accuracy must be balanced against serving cost and latency.
  • Predicting how a different-sized model in a family will perform without training or serving it, useful for planning model rollouts.
  • Selecting models for non-English or domain-specific corpora (Chinese text, code), where the paper shows rankings can shift dramatically.
  • Informing tokenizer design for long-context and test-time-scaling workloads, where token counts drive both cost and latency.
• Industry relevance: The paper frames the metric as a way to quantify inference-efficiency improvements, guide scaling decisions for future LLM development, and support deployment on heterogeneous hardware, citing the AI Flow framework's three-tier network architecture as a setting where privacy and latency matter. The authors are affiliated with the Institute of Artificial Intelligence (TeleAI), China Telecom, and the code and data are public.

Future Directions

• Balanced multilingual and multi-source training. The strong linguistic and text-source biases in mainstream LLMs point to the need for holistic training to ensure consistent performance across languages and domains.
• Better tokenizers. Given that tokenizer efficiency dominates information capacity, the paper argues tokenizers should be carefully designed to maximize compression efficiency.
• Extending beyond text. The paper notes that multimodal data can be converted into text-like representations for compression, and that large multimodal models using discrete representations are promising candidates for compressing other modalities.
• Evaluating undisclosed and post-trained models. Because many base models are not released and because RL-heavy post-training degrades compression capability, open questions remain about how to measure information capacity for closed or instruction-tuned-only models.
• Additional ablation questions. The supplied content is truncated inside the softmax temperature ablation (Figure 5), so the full conclusions of that study are not reported here.

Target Audience

Researchers and engineers working on LLM evaluation, efficiency, and inference cost; teams designing tokenizers or scaling model families; practitioners who must select among models of different sizes and architectures under compute or latency budgets; and information-theory-oriented readers interested in the compression-intelligence connection. The paper assumes comfort with entropy coding, cross-entropy, and transformer FLOP accounting, so beginners may need background reading first.

Authors’ abstract

Recent years have witnessed the rapid advancements of large language models (LLMs) and their expanding applications, leading to soaring demands for computational resources. The widespread adoption of test-time scaling further intensifies the tension between model capability and resource consumption. However, a rigorous metric that accurately reflects an LLM's inference efficiency across diverse tokenizers, parameter counts, and model architectures remains absent. Motivated by the correlation between compression and intelligence, we introduce information capacity, a measure of model efficiency based on text compression performance relative to computational complexity. A distinctive feature of information capacity is its incorporation of tokenizer efficiency, which affects inference costs but is often neglected in LLM evaluations. We assess the information capacity of 56 open-source models and observe a consistent information capacity among different-sized models within a series. Experiments on five heterogeneous datasets reveal strong linguistic biases in mainstream LLMs. Empirical results verify the accuracy of performance prediction across model sizes based on information capacity and show the correlation between information capacity and benchmark scores. This metric can be used to quantify improvements in inference efficiency and provide insights into better scaling performance for future LLM development.