Benford's Law as a Distributional Prior for Post-Training Quantization of Large Language Models

AI summary

Overview

Research area: Efficient machine learning — post-training quantization (PTQ) of large language models, with a focus on non-uniform quantization codebooks motivated by the statistics of trained weight distributions.

Technical level: Intermediate. The paper assumes familiarity with quantization basics (scales, group-wise quantization, bits, round-to-nearest, codebooks) and with standard transformer components (nn.Linear, LayerNorm, embeddings), but the core idea is explained from first principles.

Scope in one sentence: The paper shows that transformer linear-layer weights follow Benford-like leading-digit statistics while LayerNorm parameters do not, and uses that observation to build BenQ, a data-free log-spaced 4-bit quantization codebook applied selectively to transformational layers.

What This Paper Is About

Low-bit post-training quantization of LLMs is sensitive to a mismatch between the quantization codebook and the actual distribution of weights: uniform grids assume values spread evenly across the dynamic range, while neural weights concentrate near zero and often span several orders of magnitude. This paper asks whether an old regularity of natural data, Benford's Law, can serve as a lightweight distributional prior for choosing the shape of that codebook, and whether the resulting grid should be applied to every layer or only some of them.

Key Contributions

1. A diagnostic. The authors measure leading-digit (Benford-like) statistics across multiple LLM families and identify a functional dichotomy: transformational nn.Linear weights are often Benford-like, while nn.LayerNorm parameters systematically deviate.
2. A simple, data-free codebook. They introduce BenQ, a log-spaced PTQ codebook motivated by scale-broad behavior, designed as a lightweight drop-in replacement for uniform grids in group-wise PTQ.
3. Selective application. They propose and evaluate a digit-statistics-informed selective strategy that keeps stability-critical parameters (LayerNorm, and by default embeddings) in higher precision.
4. Empirical study. They evaluate 4-bit group-wise PTQ across several model families on perplexity and downstream benchmarks, and additionally report exploratory dynamic activation quantization results.

Main Findings

• A functional dichotomy in Benford adherence. Weights from transformational nn.Linear layers (attention and feed-forward blocks) closely follow Benford's distribution, whereas nn.LayerNorm weights systematically violate it, with values clustering around a single learned scalar (the paper gives 0.35 as an example). The authors hypothesize that LayerNorm weights act as learned damping factors for network stability rather than as transformations.

• Model families differ in overall adherence. Using the ε-Benford adherence test with significance level α = 0.05 and ε = 0.2, four of the six analyzed models are found to be 20%-Benford adherent, with Gemma3 1B reaching adherence as low as 12.85%. The two models that reject the null hypothesis are both from the Qwen family (Qwen-7B-Chat, minimum ε of 0.2704; Qwen-14B-Chat, minimum ε of 0.2544), which the authors attribute to their normalization layers.

• Conventional statistical tests are unsuited to LLM-scale weights. For OPT-1.3B, a Pearson χ² test gives a statistic of 23,273,831.16 with p(χ²) = 0.0000 even though the weights show only small deviations from Benford's Law, illustrating the test's sensitivity to sample size.

• Second-digit analysis preserves the dichotomy. Across all evaluated models, nn.LayerNorm layers show the highest MAD values while transformational layers stay close to the generalized Benford distribution; absolute MAD values are smaller because there are 90 two-digit categories instead of 9 digits.

• BenQ beats uniform RTN but trades wins with NF4. Under 4-bit weight-only group-wise PTQ with group size 128, BenQ consistently improves over uniform RTN. On Llama3-8B, BenQ reaches LAMBADA 0.747 and MMLU 0.629 versus RTN's 0.724 and 0.612, with perplexity 7.082 versus 7.372. Against NF4 there is no consistent superiority in either direction: differences are typically within 1–2 percentage points, and on Llama3-8B, NF4 outperforms BenQ across all metrics.

• The group-adaptive variant (BenQ GA) often performs best among the static grids. On Llama3-8B, BenQ (GA) reaches perplexity 7.028 and MMLU 0.626; on OPT-1.3B it reaches perplexity 15.753 against RTN's 16.180.

• Model families behave differently. In the Gemma3-270M results, BenQ's Lambada score is 0.253 versus RTN's 0.152, while in OPT-6.7B, Lambada is 0.665 for BenQ versus 0.661 for RTN and NF4's 0.676 is higher. On OPT-6.7B, BenQ yields the best MMLU among quantized methods at 0.256.

• ε selection matters a great deal. The paper reports that arbitrarily chosen values of ε yielded severe perplexity degradations, and that even a generally well-suited fixed value such as ε = 10⁻² failed to generalize across all model families.

• Selective exclusion is asymmetric in importance. The ablation (referenced as Table 13) shows that excluding normalization layers from quantization yields a far greater quality improvement than excluding embedding layers.

• Activation quantization is an exploratory stress test. Log-spaced grids can reduce RTN failures in some families, but the results also show that outlier handling remains essential for reliable low-bit activation PTQ.

Methodology in Plain English

The authors first run a diagnostic over pretrained models. They take every parameter tensor, compute the first significant digit of each weight, and compare the observed digit frequencies to what Benford's Law predicts, using Mean Absolute Deviation (MAD) as the discrepancy measure; they repeat the analysis for the joint first-two-digit distribution. Because χ² tests reject Benford adherence at LLM scale even for negligible deviations, they also use an ε-Benford test whose tolerance parameter makes it asymptotically independent of sample size.

They then build BenQ. Instead of evenly spaced quantization levels, it generates 2^B levels in the range [−1, 1] that are evenly spaced in the log domain, so more levels sit near zero where weights concentrate. The grid needs one number, ε, which controls how far down the log range the levels extend. Rather than tuning ε by hand, they compute it once per model from the 0.999 quantile of the sampled weight magnitudes relative to the maximum magnitude, taking the floor of the base-10 log of that ratio; sampling uses random draws from each nn.Linear weight tensor, capped at 100,000 elements. Quantization is group-wise: each block is normalized by its own maximum absolute value, then each normalized weight is mapped to the nearest grid level. Dequantization is a lookup followed by rescaling.

A group-adaptive variant (BenQ GA) skips the global grid and instead builds a log-uniform grid per group from that group's minimum and maximum. Finally, based on the diagnostic, the method quantizes only transformational layers and leaves LayerNorm in native precision, with embeddings also kept at higher precision in the main configuration. Comparisons use uniform RTN and NF4 with the same selective policy, plus GPTQ, AWQ and SINQ as stronger calibration- or optimization-based reference points. Evaluation covers perplexity on the WikiText-2 test split, LAMBADA, HellaSwag and MMLU via the lm-eval framework, across Gemma3, Qwen, Qwen3, Llama3, OPT, BLOOM, TinyLlama and DeepSeek-R1 families. Most experiments ran on a machine with an AMD Ryzen Threadripper 7960X 24-Core CPU at 5360 MHz, 256 GB DDR5 RAM and an NVIDIA H200 GPU; the group-adaptive overhead experiments used a Google Colab instance with an NVIDIA A100 GPU and 83.5 GB of RAM.

Why This Matters

Impact on research. The paper reframes a long-standing practical habit in PTQ frameworks — skipping LayerNorm and embedding layers — as a claim about weight distributions that can be measured rather than a rule of thumb. It also opens a specific avenue: using distributional statistics of weights as a diagnostic that informs quantization design, while keeping the resulting method data-free and calibration-free. The authors position BenQ as complementary to activation-aware methods (AWQ, SmoothQuant) and optimization-based methods (GPTQ, SINQ), and are explicit that these stronger baselines typically achieve lower perplexity — the goal is to understand where a simple static grid remains competitive.

Real-world applications (as implied by the paper's framing):

• Deploying LLMs on memory-constrained hardware, where 4-bit weight-only quantization reduces the memory footprint with modest accuracy loss.
• Serving pipelines that already use group-wise 4-bit quantization and want a drop-in codebook swap without adding calibration or optimization passes.
• Hardware-aware inference, since logarithmic level schedules are noted for enabling optimized operations such as bit-shifting.
• Hybrid pipelines, where a log-spaced static grid is combined with activation-aware or calibration-heavy methods.

Industry relevance. The method is data-free and requires no calibration set, which lowers the operational cost of quantizing a model compared with methods that need second-order information or activation statistics. The code is released at https://github.com/ufopcsilab/benford-quant, and the paper was reviewed on OpenReview. Because it is a codebook swap rather than a new pipeline, it is plausible as a component inside existing frameworks.

Future Directions

• Deriving provably optimal or fully adaptive strategies for selecting ε; the authors explicitly name this as an open direction, noting that the current rule is a principled but not optimal estimate.
• Extending reliable low-bit activation quantization: the exploratory results show log-spaced grids can reduce RTN failures in some families, but outlier handling remains essential, so activation-side methods remain unresolved.
• Explaining cross-family variation in Benford adherence. The authors hypothesize a link to training data quality and training procedure quality, given reported correlations between generalization and Benford compliance, but this is not settled.
• Better integrating the diagnostic with existing PTQ pipelines — for example, combining BenQ with activation-aware or optimization-based methods, and quantifying trade-offs against GPTQ, AWQ and SINQ beyond the "situating" comparisons made here.

Target Audience

Researchers and engineers working on model compression and efficient inference for large language models, particularly those interested in non-uniform quantization codebooks, group-wise PTQ, or the empirical statistics of trained network weights. It is also relevant to practitioners who want a data-free, low-overhead alternative or complement to calibration-heavy PTQ methods such as GPTQ, AWQ and SINQ. Readers with a background in numerical methods or information theory who are curious about how classical distributional laws can inform modern deep learning systems may find the diagnostic portion useful even if they do not adopt the quantization scheme.

Authors’ abstract

Post-training quantization (PTQ) is a practical way to reduce the memory footprint of large language models, but low-bit quantization is sensitive to mismatches between the quantization codebook and the empirical weight/activation distributions. We revisit Benford-like leading-digit statistics as a lightweight diagnostic of scale-broad behavior in transformer tensors. Across several model families, we observe a consistent functional dichotomy: transformational nn.Linear weights tend to be Benford-like, whereas LayerNorm parameters systematically deviate. Motivated by this observation, we propose BenQ, a data-free PTQ codebook that uses a simple log-spaced grid as a proxy for scale-broad distributions and applies it selectively to transformational layers while keeping stability-critical parameters in higher precision. In 4-bit group-wise PTQ, BenQ consistently improves over uniform RTN and trades wins with NF4 across architectures and tasks, while remaining substantially simpler than optimization-based methods. We additionally report dynamic activation quantization as an exploratory stress test: the results show that log-spaced grids can reduce RTN failures in some families, but also reveal that outlier handling remains essential for reliable low-bit activation PTQ. Code is available at https://github.com/ufopcsilab/benford-quant.