Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM

AI summary

Overview

Research area: Efficient inference for large language models — 4-bit post-training quantization (NVFP4 W4A4) of hybrid architectures that mix softmax attention with recurrent linear-attention layers (Gated DeltaNet).

Technical level: Intermediate. The paper assumes familiarity with quantization formats (4-bit weights and activations, block scaling, FP8 KV caches) and with recurrent state-space or linear-attention layers, but its central argument is presented through measurements and mechanisms rather than heavy mathematics.

Scope in one sentence: The paper builds a fully 4-bit weights-and-activations checkpoint of Qwen3.8-27B (all 496 linear layers, Gated DeltaNet included), shows it matches BF16 accuracy across six suites and retrieval to 64K, and then explains mechanistically why the recurrent half of a hybrid model is the easy half to quantize.

What This Paper Is About

Hybrid LLMs like Qwen3.8-27B devote 48 of their 64 layers to Gated DeltaNet, a recurrent operator whose fixed-size state is updated token by token. Practitioners assumed that because a recurrence carries state across tens of thousands of tokens, quantization errors would compound, so every public 4-bit build of this model left the GDN block — especially its decay and write-strength gate projections — in 8- or 16-bit precision. The authors test that assumption directly by quantizing everything to NVFP4 W4A4, including the gates, and then explain the surprising result with a four-part mechanism study of captured activations and error propagation.

Key Contributions

1. A true W4A4-GDN model, evaluated seriously. Minima quantizes all 496 linear layers of Qwen3.8-27B to NVFP4 W4A4 with GDN gates included, and under a fixed serving regime (FP8 KV cache, identical harness, per-sample validity checks) matches BF16 and two community recipes within seed noise across six accuracy suites while being the smallest and fastest at prefill.
2. A mechanism study of why it works. Activation statistics on captured 32K-token inputs, per-projection sensitivity replay, error propagation through the recurrence in lockstep FP32, and a positional decomposition of the perplexity gap together form a causal chain: block scaling localizes outliers, gate nonlinearities compress what remains, the delta rule erases state error, and the end-to-end gap shrinks with context.
3. Serving-stack findings required to measure correctly. A global-scale mismatch between per-module NVFP4 calibration and fused serving GEMMs that silently corrupts the GDN gates (and fakes better long-context perplexity); a multimodal-composite serving path that degrades long-context scoring; and a chat-template pitfall that invalidates raw-completion harnesses for thinking models.
4. A KV-cache recipe. FP8 KV cache halves KV memory and moves no task score; its one visible cost — a +0.41 perplexity penalty at 32K, three times larger for the quantized model — is eliminated by calibrated per-layer scales that are free at serving time (83% recovered, throughput unchanged within 0.4%).

Main Findings

• All quantized recipes match BF16 within seed noise. The three quantized recipes for Qwen3.8-27B span 0.54 points on the 5-task average (MMLU-Pro, GSM8K, AIME'25, GPQA-Diamond, LiveCodeBench v6), which is less than one AIME problem (3.3 points), and the largest single-task gap (RadixArk's AIME −2.5) sits inside BF16's own seed spread of 83.3–93.3. Minima matches BF16's AIME'25 score exactly at 26/30 on all four seeds.
• Generation behavior is unchanged. Minima does not "think longer": mean AIME generation is 14,531 tokens versus 14,532 for BF16, and it hits the 32K cap slightly less often (15.8 versus 16.7% on AIME).
• Quantizing the GDN block yields measurable efficiency gains. Minima is the only checkpoint whose GDN block (5.5B parameters, roughly 23% of decode weight bytes) is at 4 bits: 2.9× smaller than BF16 in VRAM and on disk (17.53 GiB versus 50.13 GiB), 7–13% smaller than either community NVFP4 build, the largest KV budget (1.81M cacheable tokens on one card), and the fastest prefill (TTFT 6.90 s to 4.03 s at 32K; +14–19% prompt throughput at 8K over Unsloth/RadixArk, whose GDN and attention GEMMs remain FP8). Decode is weight-bandwidth-bound and all three quantized models land within 4%.
• Perplexity is the honest residual. Unsloth < RadixArk < Minima at both 4K and 32K, as expected since the community recipes quantize less and pay in 1.3–2.7 GiB of weights. Minima's gap to BF16 is +0.72 at 4K but only +0.49 at 32K, it never reaches a task score, and the positional decomposition shows it is a short-context per-token effect that context washes out.
• GDN inputs are not easier than attention inputs. GDN projections read the same residual stream, carrying extreme outliers (median-layer max/RMS 63.5, kurtosis ~1,560, hot channels 100× the median channel), and 10–32% of 16-element blocks are dominated by a single value. Yet per-token A4 quantization error is uniform across every layer role (7.5–9.2%) because NVFP4's block scaling confines each outlier to its 15 neighbors; weight error (10.5–11.9%) exceeds activation error everywhere.
• The protected projections are the safest ones. In 96 per-projection replays of 8K tokens each, fully quantizing the gate projections a and b — the two tensors every public recipe keeps in BF16 — moves the layer output by only 2.1% and 2.6%, the two smallest effects, even though their own GEMM errors are 11.0% and 8.5%. The squashing in the log-space parameterization turns an ~11% pre-activation error into a 7.5% error on 1−α and a 5.2% error on β. The error Minima actually carries comes from the three plain GEMMs: out_proj (12.7%), qkv (10.4%), and z (9.9%).
• The recurrence bounds and erases noise. Running the recurrence in lockstep FP32 for 32K tokens on five layers, the full-Minima state error is flat: relS = 12.96% at token 256 and 12.31% at token 32,768 (plateau 12.6%, max 14.9%). A one-off 1% state impulse injected at t=1,024 falls to 1/e within 80–1,382 steps and to 1/10 within ~2,200–2,900, while the decay-implied horizons of the same layers reach 44K–62K tokens — the extra erasure comes from the delta rule overwriting the state along each new key direction.
• The gate parameterization is what makes gates quantization-proof. Synthetic multiplicative noise applied directly to α is devastating: 0.1% noise yields a 22% state error, because with α ≈ 1 a tiny δα is an enormous relative change in the horizon 1/(1−α). Quantizing a produces only 3.6% state error from an 11% GEMM error because the noise lands on the pre-activation, where softplus and the exponential compress it. Noise on β is harmless outright (1% noise gives 0.4% state error) since a mis-scaled correction is itself corrected by later writes.
• End to end, context washes the weight error out. The weight-quantization gap is +0.081 nats in the first half of the 32K window and +0.011 in the second; in the final 2K tokens Minima scores better than BF16 (−0.053). The FP8-KV cost behaves in exactly the opposite way — small, rising with position, and ~3× larger for Minima — the signature of an attention-path effect.
• A fused-GEMM scaling bug silently corrupts GDN gates. llm-compressor calibrates one FP32 global scale per linear module, but vLLM serves the GDN projections fused (in_proj_qkv + z as one GEMM, in_proj_b + a as another), taking the maximum of the constituent scales without rescaling the local ones. In this checkpoint the paired scales differ by 1.82× (qkv/z) and 2.75× (b/a) in every one of the 48 layers. The corrupted model is deceptively plausible: AIME drops to 80.8 versus 86.7 repaired, while 32K perplexity gets better than BF16 (a flat 6.86 versus the true 10.84) because a broken forget gate makes the state hold everything.
• Measurement pitfalls invalidate naive harnesses. Serving the multimodal composite instead of a text-only extraction changes PPL@32K (10.04 versus 10.22 for BF16). lm-eval's local-completions path sends prompts without a chat template, so "thinking disabled" never reaches the model: it opened <think> on 25–48 of 50 sampled MMLU-Pro questions, producing invalid scores of 66.3/58.6 for BF16/Minima versus the valid 80.4/79.7. The BF16 base model also scores the same tokens worse inside a 32K request than in isolated 4K windows (PPL 6.95 to 10.35) while retrieval at 64K remains 100%.
• Calibrated FP8 KV scales are free. Minima+scales adds static per-tensor FP8 scales (32 tensors on 16 attention layers) to an otherwise byte-identical recipe. PPL@32K drops from 10.84 to 10.50, recovering 83% of the penalty, with a residual +0.07 that is below BF16's own uncalibrated cost of +0.13. PPL@4K is unchanged, RULER stays 100 at 32K/64K, and throughput matches Minima within 0.4%.

Methodology in Plain English

The authors start from a serving reality: NVFP4 pairs 4-bit E2M1 values with an E4M3 scale per 16-element block, so the largest value in a block fixes the scale and an outlier degrades only its own 15 neighbors. They quantize all 496 backbone linear layers of Qwen3.8-27B (240 GDN, 64 attention, 192 MLP projections) with llm-compressor, excluding only the LM head, embeddings, convolutions, and norms, calibrating on a frozen set of 128 samples at 32K tokens. Everything is served with vLLM 0.27.1 at tensor parallelism 1 on a single RTX PRO 6000 (96 GB, SM120, native NVFP4) with an FP8 KV cache, and compared against BF16 and two public NVFP4 checkpoints that keep GDN and attention at FP8 W8A8.

To explain the result, they capture the real inputs of all 48 GDN layers while BF16 reads eight 32K-token documents, re-implement one GDN layer standalone in FP32 (verified against the reference to 6×10⁻³ median relative output difference, i.e. BF16 rounding), and inject exactly the 4-bit rounding error through fake quantization — quantize, dequantize, continue in high precision. Four experiments form a chain: activation and weight statistics at NVFP4 granularity; a per-projection sensitivity replay where exactly one projection is quantized at a time; a lockstep recurrence with one clean trajectory and eleven perturbed ones on identical inputs over 32K tokens; and a positional split of the per-token perplexity gap. Along the way they had to repair the serving pipeline — harmonizing fused global scales on the checkpoint side, extracting a text-only model, and building a chat-template harness with per-sample validity gates — before any number could be trusted.

Why This Matters

The paper reframes an architectural assumption as an empirical question and answers it with a mechanism rather than a lucky hyperparameter. It argues that the recurrent half of a hybrid LLM is the easy half to quantize, because the same gating and correction structure that makes Gated DeltaNet trainable also makes it robust to 4-bit rounding. It also documents a serving-stack bug that is invisible on checkpoints which happen to keep fused-adjacent modules at equal scales, but which corrupts any recipe that quantizes the GDN block.

Real-world applications:

• Single-GPU serving of large models. Minima fits in 17.53 GiB of weights with a 1.81M-token KV budget on one card, versus BF16's 50.13 GiB, making a 27B-class hybrid deployable on commodity accelerator memory.
• Faster prefill for long prompts. TTFT at 32K drops from 6.90 s to 4.03 s versus BF16 with +14–19% prompt throughput over the community FP8-GDN builds — directly relevant to retrieval-augmented and document-processing workloads.
• Long-context agentic and reasoning serving. Accuracy holds on AIME'25, GPQA-Diamond, and LiveCodeBench v6 while truncation at the 32K cap stays at 16.7/14.6% for BF16 and 15.8/14.6% for Minima, so generation length does not inflate.
• KV-cache sizing under fixed hardware. FP8 KV cache grows capacity 1.8–1.9× with no task-score movement, and calibrated scales remove the residual perplexity cost — the recipe the authors release unconditionally.

Industry relevance: The practical guidance is short and immediately actionable for anyone deploying hybrids: quantize everything including the recurrent block, serve FP8 KV, and ship calibrated per-tensor scales. The paper also shows that quantization-aware training is not necessary — calibration-only post-training quantization reaches BF16-level task accuracy on all 496 layers — which matters for teams that cannot afford a distillation or QAT pipeline.

Future Directions

• Beyond 128K context. The evidence covers 32K-token perplexity and 64K retrieval; the bounded-error mechanism predicts longer contexts, but the authors state explicitly that 128K+ behavior is extrapolation and was not stress-tested.
• Other architectures and parameterizations. The gate-shielding argument depends on the log-space softplus/exponential parameterization; recurrent mixers with linearly-parameterized decay may not enjoy it, since 0.1% direct noise on α demonstrably harms the state.
• Comparison against quantization-aware training. QUASAR's concurrent NVFP4 checkpoint of the same model learned its 4-bit weights by distillation from the BF16 teacher and appeared after this measurement campaign closed; a controlled head-to-head under a shared harness is left open, along with whether QAT adds anything over calibration-only PTQ.
• Sub-4-bit and kernel overhead. Concurrent work in the authors' group extends the recipe to the full model (embeddings, LM head, multi-token-prediction head at NVFP4 W4A4)

Authors’ abstract

Hybrid LLMs pair softmax attention with linear-attention layers such as Gated DeltaNet (GDN), whose recurrent state summarizes the context in fixed size. Early community 4-bit quantizations of Qwen3.8-27B (48 GDN layers, 16 attention layers) left the GDN block in 8- or 16-bit precision -- especially its decay and write-strength gates -- on the intuition that errors in a recurrence accumulate over long contexts. We test that intuition by building Minima: NVFP4 W4A4 on all 496 linear layers, GDN included. Across perplexity at 4K/32K, MMLU-Pro, GSM8K, AIME'25, GPQA-Diamond, LiveCodeBench, and RULER retrieval to 64K, Minima matches BF16 within seed noise (5-task average -0.52) while being the smallest (17.5 GiB) and fastest-prefill (+14-19%) recipe we compare, and its 32K perplexity gap shrinks with position. A four-part mechanism study explains why: (i) NVFP4's 16-element block scaling localizes the residual stream's extreme outliers, equalizing activation error across layer roles; (ii) the supposedly fragile gate projections are the least sensitive -- softplus/exponential and sigmoid parameterizations compress ~11% GEMM error to ~2% output error; (iii) the delta-rule recurrence holds injected noise at a flat plateau over 32K tokens and forgets a state impulse within hundreds of steps, because each write overwrites the state along the current key direction; (iv) the per-token quantization cost washes out with context instead of compounding. We also repair a global-scale mismatch that arises when per-module-calibrated NVFP4 checkpoints are served by kernels that fuse those modules into one GEMM, and show calibrated FP8 KV-cache scales are performance-free. The result: a practical recipe -- quantize everything, ship KV scales -- and a mechanistic account of why the recurrent half of a hybrid LLM is the easy half to quantize. Checkpoint: https://huggingface.co/minima-ai/mnma_qwen3.8_27b_nvfp4