PACE: Periodic and Channel-Aligned Encoding of Four-Bit Recurrent States

AI summary

Overview

Research area: Efficient inference for linear-attention language models — specifically post-training quantization of the fixed-size recurrent state that replaces the key–value cache in gated-delta models.

Technical level: Intermediate. The paper assumes familiarity with attention, quantization, and the idea of a recurrent state, but its central argument is intuitive: round the state less often and align quantization groups with the structure of the state.

Scope: A post-hoc, training-free four-bit state quantizer for gated-delta hybrid models, evaluated on four backbones from 4B to 35B parameters across five benchmarks, plus state-drift and decoding-throughput measurements.

What This Paper Is About

Linear-attention models keep their whole context in a recurrent state of fixed size, and a serving system must store one copy of that state for every sequence it decodes simultaneously. Storing the state at lower precision would let more sequences share one accelerator, but existing state quantizers round the state after every generated token and report that four bits lose accuracy unless the bit width is raised or the model is partly fine-tuned. PACE asks whether changing when the state is rounded — once every few tokens rather than every token — plus how the scale groups are shaped, can make four-bit states viable without any training.

Key Contributions

1. A rounding-error bound tied to the rounding period. The authors derive that the error energy a head retains is E_h = e_h / (1 − a_h^(2·dP)), where a_h is the head's mean decay and dP is the decode period, and pair it with a memory accounting that counts the decode buffer, not just the stored state. On four backbones, rounding every eight generated tokens reduces the accuracy loss at every bit width below eight.

2. The observation that large state entries form columns. The large entries of the gated-delta state concentrate at the same value index across key indices, and permutation experiments show this comes from channels of the delta value u_t that stay large across tokens. This motivates scale groups taken along the key axis instead of the value axis.

3. PACE itself, a four-bit state quantizer with channel-aligned scale groups and head-level precision allocation, in two variants (PACE-4 and PACE-4/8), compared against DAMP and a reimplementation of Q-Mamba's decoupled scale quantization on four hybrid models from 4B to 35B parameters.

4. Triton kernels for periodic decoding with a packed four-bit state, showing that rounding every few tokens preserves the decoding speedup that the decode buffer provides.

Main Findings

• Rounding less often helps at every sub-eight-bit width. Using a prior-work quantizer (uniform round-to-nearest, value-axis groups of 32) at four to eight bits with dP ∈ {1, 8, 32}, per-token rounding loses more accuracy as the width falls, and rounding every eight tokens scores above per-token rounding at every width below eight bits on all four backbones. A period of eight closes most of the gap that a period of 32 closes, at a quarter of the buffer memory.

• A residual gap remains on the larger models. On the three larger backbones a gap to full precision persists at every width, concentrated in IFEval and RULER-32k; the RULER-32k gap is similar at all three decode periods, which the authors suggest means that part of the gap arises during prefill.

• Large state entries cluster in columns. The ratio of a group's largest absolute entry to its root mean square (max/RMS) is larger for value-axis groups than key-axis groups at every group width, and the difference grows with width. At g = 128, the value-to-key ratio is 1.588, 1.555, 1.601 and 1.871 on the 4B, 9B, 27B and 35B-A3B models respectively. The share of large entries falling in the heaviest 5% of columns is 82.6%, 78.6%, 83.9% and 87.8%.

• The columns come from the delta-value channels. Permuting the channels of u_t at every token flips the ratio below one (0.776, 0.760, 0.808, 0.963 at g = 128) and drops the outlier mass to 35.5%, 34.5%, 37.1% and 45.4%; permuting the key entries instead widens the ratio (2.265, 2.220, 2.261, 2.580) and raises outlier mass to above 92%.

• Key-axis groups beat value-axis groups in ablations. At 5.28 resident bits per value, key-axis groups of 32 give IFEval of 80.59 ± 2.53 on Qwen3.5-4B and 78.37 ± 2.84 on 27B, with WikiText-103 perplexity of 7.719, versus 73.75 ± 3.29 / 75.05 ± 3.24 and perplexity 8.158 for value-axis groups at the same width. Q-Mamba DSQ at 5.03 bits gives 74.86 ± 3.22 / 73.94 ± 3.33 and perplexity 8.320, and key-axis groups of 64 at 5.03 bits give 80.59 ± 2.73 / 78.93 ± 2.78 with perplexity 7.734.

• Head-level allocation beats channel-level protection. At 6.44 resident bits, head allocation on the key axis gives IFEval of 81.52 ± 2.51 (4B) and 78.00 ± 2.88 (27B) with perplexity 7.687, versus DAMP INT4's 79.48 ± 2.68 and 79.85 ± 2.91 with perplexity 8.003 — noting DAMP INT4 is better on the 27B IFEval score.

• PACE-4/8 is competitive with DAMP INT8 at lower memory. Across the four backbones, PACE-4/8 (int8 buffer, 6.44 resident bits) holds 0.69 of DAMP INT8's resident bits. Averages are 80.8 vs 80.9 (4B), 81.4 vs 83.7 (9B), 84.5 vs 87.4 (27B) and 78.3 vs 86.3 (35B-A3B), with full precision at 80.8, 84.5, 87.5 and 86.5. PACE-4/8 scores below DAMP INT8 on IFEval on the three larger backbones and close on average, with the other benchmarks differing in both directions; on Qwen3.5-4B its average is lower, with lower GSM8K, MMLU-Pro and IFEval.

• PACE-4/8 beats DAMP INT4 at equal memory. At the same 6.44 resident bits, PACE-4/8 with an int8 buffer scores above DAMP INT4 on every backbone, with the largest difference on Qwen3.6-35B-A3B (78.3 vs 69.0 average).

• PACE-4 beats Q-Mamba DSQ at equal memory. At 5.03 resident bits, PACE-4 scores higher than Q-Mamba DSQ on every backbone (79.6 vs 75.2, 80.0 vs 72.8, 84.2 vs 78.9, 77.3 vs 71.2).

• State drift falls as the period grows. On Qwen3.5-4B over the first 512 tokens of the WikiText-103 test set, relative state error decreases with dP from 1 to 64 at every token position for three four-bit quantizers, and drift grows more slowly than the number of roundings.

• Periodic rounding keeps the decoding speedup. On Qwen3.5-4B on one H100 PCIe GPU at a 512-token context, periodic INT4 decoding has a lower time per output token than per-token bf16 from batch size 16 upward, with the gap growing with batch size; most of the reduction comes from the decode buffer, and the INT4 state lowers TPOT slightly further.

• Buffered values resist four-bit rounding less than buffered keys. Rounding only the buffered values to four bits collapses IFEval to 13.9 / 22.7 / 37.0 / 12.2 on the four models, while rounding only the keys gives 78.2 / 82.4 / 86.5 / 80.4. The explanation offered is that the kernel L2-normalizes buffered keys after rounding, so only the direction of a key error reaches the state.

Methodology in Plain English

The authors start from an observation about how chunked kernels already work. During prefill, the kernel processes the prompt in chunks and only writes the state at each chunk boundary, so every published quantizer already rounds the prefill state once per chunk. During decoding, the kernel is called once per token, so the state gets written and rounded after every single token. PACE borrows the prefill behaviour for decoding: it holds the tokens generated since the last rounding in a small decode buffer, lets the state run unrounded over them, and then dequantizes, advances, and requantizes once per flush. Fewer roundings means fewer injections of rounding noise, and the decay term in the gated delta rule washes out old error between flushes.

The second idea comes from looking at what the state actually looks like. When one value channel is consistently large across many tokens, the outer-product writes pile up in a single column of the state matrix. A quantizer that groups entries along the value axis therefore mixes big and small entries in one scale group, and the big entries set a coarse step size for everything else. PACE instead groups entries along the key axis so each group stays inside one column, and stores one fp16 scale per group.

The third idea is to spend bits unevenly. Since the retained error is a per-head quantity that depends on the head's decay, the authors measure each head's rounding error once per checkpoint on eight held-out windows of 2,048 tokens and give eight bits to the quarter of heads per layer with the largest retained error, keeping the rest at four.

Two variants follow: PACE-4 is uniform four-bit with symmetric key-axis groups of 64 and an int8 buffer; PACE-4/8 adds head allocation, uses key-axis groups of 32 with an integer zero point, and holds buffered keys at four bits with values in bf16. The memory metric — resident bits per value — counts the stored state and the buffer together, divided by the number of state values, with the buffer adding c · dP bits per value for a constant c derived from key and value head dimensions.

Evaluation uses Qwen3.5-4B, 9B, 27B and Qwen3.6-35B-A3B with the LM Evaluation Harness and greedy decoding, on GSM8K, MMLU-Pro (a subset of 40 questions per subject, n = 560), IFEval (n = 541), RULER variable tracking at 32,768 tokens, MATH-500, and WikiText-103 perplexity. All generative tasks are zero-shot with the chat template and thinking disabled.

Why This Matters

For research, the paper reframes state quantization as a question about when to round rather than only how. The error bound makes the decode period a tunable knob with a principled justification, and the column-structure finding ties quantizer design to a specific property of the gated delta update — both are transferable observations that other quantizer designs can adopt independently of PACE.

Practical implications include:

• Higher serving density. At 64 concurrent sequences, fp32 states already take 10–41% of the memory that weights and states occupy together; cutting state memory means more sequences per accelerator.
• Long-context inference on fixed hardware. RULER-32k results target the regime where per-sequence state memory becomes the binding constraint.
• Memory-constrained or on-premises deployment, where a training-free, post-hoc quantizer can be applied to an existing checkpoint without fine-tuning part of the weights.
• Batch-heavy workloads such as agentic or retrieval pipelines that decode many sequences at once, where the throughput measurements show periodic INT4 decoding beating per-token bf16 from batch 16 upward.

For industry, the appeal is that PACE requires no training, keeps the kernel structure that chunked implementations already have, and ships with Triton kernels — all of which lower the barrier to adopting four-bit states in an existing serving stack.

Future Directions

• Vary the prefill period. The prefill period is fixed at the kernel's 256-token chunk in every benchmark evaluation, and the authors note that the persistent RULER-32k gap appears at all decode periods, suggesting a prefill-side origin; the effect of rounding the state more or less often during prefill is not measured.
• Explore the wider configuration space. The paper covers only part of a space spanning prefill and decode periods, state and buffer bit widths, group axis and width, and the fraction of heads held at eight bits.
• Test other quantizers under periodic rounding. Only PACE, DAMP INT4 and Q-Mamba DSQ are evaluated with a period above one; static scales, stochastic rounding, and rotations of the state are untested under periodic rounding.
• Validate beyond the gated delta rule. The head decays, the column structure, and the head ranking are measured only on four Qwen3.5/Qwen3.6 hybrids; whether they carry over to other families, to fully recurrent models, or to token mixers with different update or decay rules is open. The authors note periodic rounding in principle applies to chunked gated linear attention, Mamba-2 and DeltaNet variants.

Target Audience

Researchers and engineers working on efficient inference for linear-attention and hybrid language models, particularly those implementing state quantization or serving systems that must budget per-sequence memory. It will also interest readers following the broader line of work on KV-cache and recurrent-state compression, and practitioners deciding whether a four-bit state is usable without fine-tuning. Some background in quantization and recurrent formulations is assumed.

Authors’ abstract

Linear-attention language models keep their context in a recurrent state of fixed size, where an inference system stores one copy of this state for every sequence it decodes at the same time. Storing the state at lower precision therefore lets more sequences share an accelerator. Existing state quantizers round the state after every generated token and report that four bits lose accuracy unless the bit width is raised or the model is fine-tuned.\par We introduce PACE, a post-hoc four-bit state quantizer. PACE rounds the state only once every few generated tokens, as chunked kernels already do once per chunk during prefill, and keeps the tokens generated in between in a small buffer. Rounding less often reduces the error that accumulates in the state. Additionally, we find that the large entries of the gated-delta state gather in columns, because some channels of the update written at each token remain large across many tokens. PACE therefore shares each quantization scale within one column. It also keeps the heads that retain rounding error longest at eight bits. On four hybrid models from 4B to 35B parameters, PACE with mixed four- and eight-bit heads is competitive with eight-bit DAMP on the three larger models at about two thirds of its memory, PACE with four-bit heads throughout scores higher than Q-Mamba at equal memory, and rounding every few tokens preserves the decoding speedup of the buffer.