Float8@2bits: Entropy Coding Enables Data-Free Model Compression

AI summary

Overview

• Research area: Post-training model compression / quantization for large language models, bridging quantization with lossless entropy coding.
• Technical level: Intermediate. Readers should know roughly what quantization bit-width and perplexity mean, but the paper's core idea (compress less aggressively, then squeeze the redundancy out losslessly) is intuitive.
• One-sentence scope: The paper introduces EntQuant, a calibration- and training-free compression framework that keeps weights in an 8-bit format, optimizes them for low entropy, and losslessly entropy-codes them to reach effective 2-bit storage while preserving model quality.

What This Paper Is About

Post-training compression is split between fast, data-free methods (NF4, HQQ) that break down below 4 bits, and data-hungry methods (GPTQ, OmniQuant, QuIP#, EfficientQAT) that recover quality but need calibration sets or long recovery training. The core problem is that compression rate is rigidly tied to bit-width: an 8× compression forces a 2-bit format, leaving only four distinct weight values to represent a complex distribution. EntQuant asks whether entropy coding can break that coupling — keeping high-precision weight representations (Float8/Int8) for expressivity while achieving 2-bit storage by making the weights statistically compressible.

Key Contributions

1. Calibration-free extreme compression. EntQuant reaches effective rates down to ~2 bits per parameter with no calibration data and no recovery training, classifying it as a pure "Level 1" method in the standard quantization taxonomy — uniquely suited to instruction-tuned or reasoning models whose training data is inaccessible or legally restricted.
2. Decoupling compression rate from bit-width. By minimizing weight entropy within an 8-bit format and losslessly coding the result with GPU-accelerated Asymmetric Numeral Systems (ANS), the method supports arbitrary effective bit-rates while retaining the full dynamic range of the base format at inference time.
3. Simplified outlier handling. Instead of explicit outlier detection or complex grouping schemes, EntQuant uses only channel-wise scaling, letting the entropy optimization naturally concentrate precision where it matters. A simple "super weight" layer-exclusion trick further helps Int8 and also improves baselines like NF4 and HQQ.
4. High-speed optimization and practical inference. Compression takes seconds per layer (under 10 minutes for a 70B model on an H100), and a block-wise on-the-fly decoding pipeline adds only 1.5–2× latency over the uncompressed BFloat16 baseline — matching NF4 inference speed.

Main Findings

• Data-free baselines collapse at 2 bits; EntQuant does not. On LLaMA base models, HQQ at 2 bits produces perplexities in the hundreds to thousands (e.g., 1.4e3 to 3.0e4 on various models), while EntQuant at 2.1 bits stays in a functional range (C4 perplexity 6.47 on LLaMA-2 70B, 9.90 on LLaMA-2 7B).
• Competitive with methods that use data and heavy compute. On LLaMA-2 70B at 3 bits, EntQuant scores C4 5.74 and LM Eval 71.7 versus base model 5.52/72.8 — edging out GPTQ (5.85/71.5), OmniQuant (5.85/71.1), and EfficientQAT (5.71/71.8), and close to QuIP# (5.67/72.1). At 2.1 bits, EntQuant (6.47/68.6) is comparable to QuIP# (6.12/70.9) and EfficientQAT (6.48/68.9), which require ~50h and ~41h of computation respectively.
• Higher representational expressivity at lower storage. A 2-bit EntQuant model has ~34.6 unique values per weight matrix on average, more than a 4-bit fixed bit-width model's 16 — direct evidence that storage cost and precision have been separated.
• Strong performance on instruction-tuned models. Evaluated across 16 open-weight LLMs (LLaMA-1/2/3.1/3.3, Qwen3, OLMo 3.1 Instruct, Mistral Large Instruct 24.11) with over 480 runs, EntQuant holds up on IFEval, GSM8K CoT, GPQA, and MMLU, where calibration-based methods often degrade alignment or reasoning behavior.
• Inference overhead is modest and amortizes. EntQuant is 1.5–2× slower than BFloat16, roughly on par with NF4, and faster than HQQ. Because weights are decompressed once per transformer block per forward pass, the relative overhead shrinks as context length grows (tested up to 8192 tokens on LLaMA-3.1 70B). It is also roughly 3× (prefill) to 45× (decode) faster than CPU offloading.
• Memory footprint is dominated by compressed weights. For LLaMA-2 70B at 2.1 bits: ~18.8 GiB compressed weights, ~0.8 GiB decompression buffer, ~1.25 GiB KV cache; scales and ANS metadata add under 5% overhead. A 70B model can fit on a consumer 32 GiB GPU at 3 bits and below.
• Below ~2 bits, breakdown. Compressing significantly below 2 bits causes performance collapse, particularly for smaller base models, and smaller models generally degrade more under compression than larger ones.
• Float8 beats Int8 by default. Int8 is viable but more sensitive to rare "super weights" in early down-projection layers; excluding those layers recovers Int8 performance and modestly helps Float8.
• Activation quantization adds a small penalty. In W8A8 configurations, EntQuant shows a slight drop versus W8A16, with LLaMA-2 70B showing a somewhat larger gap.

Methodology in Plain English

The method works in three stages.

First, quantize mildly but in a high-precision format. Rather than crushing weights into 2-bit integers, EntQuant rounds them into a Float8 (or Int8) grid using symmetric per-channel scaling — one scale factor per output channel, which costs almost nothing in memory or inference time.

Second, optimize the weights to be statistically boring. The goal is to make the distribution of quantized values as concentrated (low-entropy) as possible, so that entropy coding can store the matrix in few bits. Directly minimizing entropy is non-differentiable, so the authors use a tractable proxy: a rate-distortion objective combining a relative ℓ1 reconstruction error with an ℓ1 penalty on the weights, which serves as a differentiable stand-in for entropy. They solve this per layer with L-BFGS in PyTorch, tuning only the scale parameters (not every weight), using a straight-through estimator to push gradients through the rounding operation. The trade-off parameter λ maps log-linearly to the target entropy and is largely model-independent, making it easy to select. Because only the weight matrix is required as input, the whole procedure is data-free.

Third, losslessly compress and decode on the fly. The quantized matrix is flattened into a symbol stream and encoded with an ANS coder (NVIDIA's nvCOMP), producing a compact bitstream. At inference, all weights of a transformer block are jointly decompressed into a per-device buffer immediately before that block's forward pass; individual layers are accessed as tensor views with no copying, and the buffer is overwritten by the next block. GEMM computation builds on the Float8 Marlin kernel. The compressed bitstream, a small set of scales, and ANS metadata are all that need to be stored.

The framing is deliberately classical: it mirrors the JPEG pipeline, where a signal is quantized and then losslessly coded, except here the "signal" is a weight matrix and the codec runs on a GPU inside the inference loop.

Why This Matters

This work suggests that fixed bit-width quantization, which has been the dominant paradigm in LLM compression, is approaching a barrier that entropy coding can route around. By treating precision and storage as separable concerns, it reframes extreme compression as an information-theoretic optimization problem rather than a data-fitting problem — and it achieves results competitive with fine-tuning-based methods at a fraction of the cost, without any data.

Real-world applications:

• Self-hosting large models on constrained hardware. A 70B model compressed to ~2 bits fits within consumer GPU memory, enabling private, low-latency deployment without an API.
• Compressing models whose training data cannot be shared. Instruction-tuned, reasoning, and domain-specialized models with proprietary or legally restricted corpora can be compressed without any calibration set.
• Regulated industries. Healthcare and finance deployments where GDPR-style rules prevent repurposing sensitive data for calibration pipelines.
• Rapid iteration on new model releases. With frontier models shipping weekly, a sub-10-minute compression step replaces multi-hour calibration pipelines as a routine part of the deployment workflow.
• Safety-tuned model preservation. Avoiding calibration sidesteps documented risks that calibration degrades alignment and reasoning behavior unpredictably.

Industry relevance is direct: the compression is fast, hardware-portable (ANS is commoditized, with hipANS for AMD and nvCOMP for NVIDIA), requires no specialized kernels or custom group structures, and the code is publicly released.

Future Directions

• Fused decoding kernels. EntQuant currently builds on existing Float8 Marlin GEMM kernels with a separate decompression stage. The paper notes that custom fused kernels could close the remaining gap to uncompressed BFloat16 inference, following the historical trajectory of GPTQ and QuIP.
• Richer quantization and entropy proxies. The authors deliberately chose the simplest possible instantiation — ℓ1 regularization, channel-wise scaling only. More sophisticated quantization schemes and better differentiable entropy surrogates are obvious extensions.
• Broader evaluation. The study used a 5K GPU-hour budget across 16 models; larger models, mixture-of-experts architectures, and more real-world tasks remain untested.
• Beyond language models. Since EntQuant operates on weight matrices alone, it is architecture-agnostic and could apply to diffusion models and other foundation model families.
• Combining with activation quantization. W8A8 results show a modest penalty, and the missing fused W8A8 kernels mean potential speedups there are unmeasured.

Target Audience

Machine learning engineers and researchers working on model compression, efficient inference, and LLM deployment, particularly those who need to compress instruction-tuned or proprietary models without access to calibration data. The paper is also useful for practitioners choosing a quantization method for self-hosted inference on memory-constrained hardware, and for information-theory-adjacent researchers interested in reviving classical rate-distortion and entropy-coding techniques inside modern deep learning pipelines. Readers without background in quantization or entropy coding will need to consult the preliminaries, but the central argument is accessible.

Authors’ abstract

Post-training compression is currently divided into two contrasting regimes. On the one hand, fast, data-free, and model-agnostic methods (e.g., NF4 or HQQ) offer maximum accessibility but suffer from functional collapse at extreme bit-rates below 4 bits. On the other hand, techniques leveraging calibration data or extensive recovery training achieve superior fidelity but impose high computational constraints and face uncertain robustness under data distribution shifts. We introduce EntQuant, a framework that unites the advantages of these distinct paradigms. By matching the performance of data-dependent methods with the speed and universality of data-free techniques, EntQuant enables practical utility in the extreme compression regime. Our method decouples numerical precision from storage cost via entropy coding, compressing a 70B parameter model in less than 10 minutes. We demonstrate that EntQuant does not only achieve state-of-the-art results on standard evaluation sets and models, but also retains functional performance on more complex benchmarks with instruction-tuned models, all at modest inference overhead.