The Structure of Quantization Damage in LLMs: Why the Next Bit Should Be Spent Globally

AI summary

Authors: Jundong Hu, Shekar Ramachandran (PayPal AI) arXiv: 2609.01587v1 [cs.LG], 01 Sep 2026

Overview

• Research area: Efficient large language model inference — specifically post-training quantization (PTQ), mixed-precision allocation, and the interpretability of quantization damage.
• Technical level: Intermediate. The paper assumes familiarity with 4-bit quantization schemes (RTN, GPTQ, AWQ, group-wise scaling) and with layer-level intervention experiments, but its central argument is a resource-allocation rule that can be read without deep optimization background.
• Scope: An empirical study of where 4-bit quantization damage lives across 9 open-weight models in 4 architecture families, and how a small additional precision budget should be spent to repair it.

What This Paper Is About

Quantization makes LLMs cheaper to serve, but the accuracy cost is uneven — some models and tasks lose far more than others under the same 4-bit scheme, and practitioners currently mitigate this by trial and error. The paper asks two practical questions: where does quantization damage actually occur, and how should a limited precision budget be allocated to repair it. The authors test three intuitive hypotheses about where the damage lives, then compare two competing ways of spending the same small bit budget.

Key Contributions

1. A resource-allocation rule for precision. At matched effective bits/weight, spending the budget globally on finer quantization granularity beats oracle-selected local repair of the most-recoverable whole layers for all 8 group-128-compatible models, by 21–52 points — including Qwen3-8B, the most concentrated model. The rule is stated as unconditional within the tested budget setting.
2. No cheap localizer predicts the marginal value of precision. Task-circuit drift, the causal computation site (activation patching), and weight/reconstruction statistics all fail to locate where restoring precision recovers accuracy. Only the causal mixed-precision intervention identifies those locations.
3. A structural description of recovery. Damage is diffuse (roughly half the layers), the residual is budget-limited (8-bit is near-lossless across RTN, GPTQ, and AWQ), and where recovery does concentrate, the site is architecture-correlated within a family — leave-one-out predicts the held-out LLaMA-3.x size, 3/3 — but not across families.

Main Findings

• Three plausible localizers all fail the causal test. H1 (damage follows task circuits): the raw drift–damage correlation is r = +0.377 across 198 task–model pairs, but this is a task-type confound that falls to +0.05 (not significant) after jointly controlling for model and task category, and is TOST-equivalent to zero within ±0.2. H2 (damage lives where the model computes): causal activation patching places computation at the boundary layers (first and last MLP), but protecting that boundary pair recovers ≤13% of the gap in 6 of 9 models — only Mistral-7B is substantially helped, at 55.5%, with Qwen3-1.7B and OpenLLaMA exceeding 13% only modestly. H3 (weights reveal the vulnerable layer): a weight-std "fragility" heuristic finds the recoverable layer only within the LLaMA family, missing Mistral's endpoint and Qwen's L4, while reconstruction error is negatively related to recovery in Qwen3-8B, whose most-recoverable layers are among the lowest-error ones.

• Damage is diffuse. Ranking layers by their own protect-one recovery (an oracle ordering) and restoring the top-k to 8-bit, reaching 50/75/90% of the gap takes roughly 20/49/73% of layers on average (9/9 models). No single layer accounts for more than approximately 44% of the damage in any model. Recovering most of the gap needs roughly half the network, not a few layers.

• One model is the exception. Qwen3-8B is sharply concentrated: its 3 highest-recovery layers recover essentially the whole gap, though its single best layer recovers only approximately 40%. Concentration is therefore family-contingent rather than a general property.

• Peak recovery location tracks architecture within a family, not across. LLaMA-3.x peaks at the same layer (L1) at every size, so leave-one-out predicts the held-out size 3/3. Mistral (n = 1) peaks at its last layer; Qwen peaks at its own layer only at 8B. Qwen3's peak moves with scale, and no signal forecasts a new family's site before a sweep.

• At a matched budget, global granularity wins everywhere. Starting from per-row 4-bit RTN, a +0.146 effective bits/weight increment can be spent globally (finer granularity everywhere, per-row to group-128 scales) or locally (restore the most-recoverable whole layers to 8-bit). Global wins for all 8 group-128-compatible models by 21–52 points, and the local arm is oracle-selected, so this is a conservative bound.

  Model	global (g128)	local @ matched	top-1 conc.
  Llama-3-8B	78.8	44.7	43.7
  Qwen3-8B	76.6	54.9	39.7
  Mistral-7B	72.9	51.7	50.9
  Qwen3-1.7B	66.9	14.9	14.9
  Llama-3.2-1B	64.9	17.2	29.5
  Llama-3.2-3B	62.9	36.7	36.5
  Qwen2.5-0.5B	41.1	3.4	3.9
  Qwen3-0.6B	33.4	4.8	4.7

• Concentration does not rescue local repair. The local arm funds only approximately 1.3 layers, and even Qwen3-8B's single best layer recovers just approximately 40%, so a model whose top 3 layers would close the gap cannot assemble enough of them at this budget. Concentration is a property of the cumulative curve, not of what a tight budget can fund.

• The comparison is insensitive to large bit-accounting errors. For the 7 diffuse models, local repair would need 5.16–6.33 effective bits/weight to match global — 7–15× the disputed +0.146 increment. Only Qwen3-8B is close: local ties global at 4.206 bits/weight (+0.05 over g128), and its local arm is oracle-selected.

• Task-bootstrap robustness. Across 5,000 bootstrap resamples of the approximately 22 CORE tasks, P(global > local) ≥ 0.95 for 8/8 models, and the 95% margin CI excludes the local arm for 6/8 (the exceptions are the near-noise 0.5B model and the smallest-margin singleton).

• Granularity, not the algorithm, drives the 4-bit gain. Per-row to g128 RTN recovers +0.095 CORE on average, while GPTQ and AWQ add only +0.020 and +0.017 over the same granularity (0.21× and 0.18× the granularity gain). Method-isolated recovery is approximately 34% (GPTQ) and approximately 28% (AWQ) of the fp16 to g128-RTN gap. Effective bits per weight: per-row RTN 4.01, g128 4.156, fp16 16.

• 8-bit is near-lossless. At 8-bit every lever collapses: per-row RTN matches fp16 within harness noise (mean gap to fp16 −0.001, every model within ±0.005; residual CI contains zero for 6/8 models), and the Qwen3-8B GPTQ anomaly above fp16 resolves at 8-bit (GPTQ-8 0.418 vs fp16 0.411).

• Circuits cluster by task demands, not surface labels. Across all 9 models the 22 tasks recur in 5 groups, with the strongest cut being formal/structured vs. the rest (drawn identically by every model across a 16× size range). Head-only silhouette at k = 5 ranges 0.48–0.60, and head-only clustering beats head + MLP in 8/9 models.

Methodology in Plain English

The researchers first needed a trustworthy answer to "which layers matter?" rather than relying on correlation. They built one by causal intervention: take a model quantized to 4-bit, raise a single layer (or a chosen set) to 8-bit, leave everything else at 4-bit, and measure how much of the 4-bit-to-8-bit accuracy gap comes back. This "marginal value of precision" is the ground truth. They then checked whether three cheap signals could predict that ground truth: (H1) how much a task's attention-head activation pattern drifts under quantization, (H2) which layers the model's prediction causally depends on, measured by activation patching, and (H3) per-layer weight statistics such as standard deviation and reconstruction error.

Damage was measured as each model's fp16-to-4-bit CORE accuracy gap, evaluated on 22 tasks at 200 samples per task under a fixed continuation-scoring harness with seed 1337. The 9 models span 4 architecture families and a 16× size range: LLaMA-3.2-1B/3B, LLaMA-3-8B, Qwen2.5-0.5B, Qwen3-0.6B/1.7B/8B, Mistral-7B, and OpenLLaMA-3B.

For the allocation experiment, the authors compared two ways of spending the same +0.146 effective bits/weight of extra precision: globally, by switching from per-row scales to group-128 scales everywhere, or locally, by restoring the highest-recovery whole layers to 8-bit. They matched the arms on effective bits/weight using a first-order, weight-only accounting with interpolated fractional layer counts, and scored both as a percentage of the per-row RTN4-to-RTN8 CORE gap. They also ran a per-model 4-bit ladder — per-row RTN, g128 RTN, GPTQ, AWQ, 8-bit RTN, fp16 — using llm-compressor 0.6.0.1 (W4A16, group-128) to separate granularity effects from algorithm effects.

One practical constraint: the group-128 configuration requires a width divisible by 128. OpenLLaMA's intermediate dimension is 8640, which is not divisible by 128, so it is excluded from the budget analysis, leaving 8 models.

Why This Matters

Impact on research. The paper draws a sharp distinction that is easy to miss: a cheap signal can correlate with quantization damage without identifying where restoring precision actually improves accuracy. This undercuts a broad family of heuristics — sensitivity-based allocation, salience-based weight protection, task-aware quantization — that assume damage is localizable. The authors are explicit that they do not test Hessian sensitivity itself, and that their "no few-layer fix" claim is scoped to whole layers at greedy, recovery-ranked layer granularity, leaving weight-level protection a separate

Authors’ abstract

Post-training quantization (PTQ) is widely used to reduce the cost of serving large language models (LLMs), but its accuracy cost is uneven and is often tuned per model. We study where quantization damage occurs and how to allocate a small additional precision budget. Using causal mixed-precision intervention as ground truth (raise each layer to 8-bit in turn and measure the accuracy it recovers) across 9 open-weight models in 4 architecture families, we test 3 intuitive hypotheses: that quantization damage lives in task circuits, where the model computes, or in weight statistics. None of them predicts which layers benefit from restored precision. Recovery is instead diffuse: for 8 of 9 models, recovering 75% of the gap takes roughly half the layers; the lone exception, Qwen3-8B, is sharply concentrated. At a matched precision budget, spending it globally on finer quantization granularity beats locally repairing the most recoverable layers for all 8 group-128-compatible models (all but OpenLLaMA, whose width rules out group-128), by 21-52 points, including the concentrated Qwen3-8B. We report 2 secondary findings: the residual is budget-limited (8-bit is near-lossless in our evaluation across RTN, GPTQ, and AWQ), and the location of peak recovery correlates with architecture within a family, though not across families. Within this budget setting, global granularity is a better default than selectively protecting critical layers. More broadly, cheap signals that correlate with quantization damage do not necessarily identify where restoring precision improves accuracy; this must be tested with causal intervention.