ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inference

AI summary

Overview

• Research area: Post-training quantization (PTQ) for large language models, spanning quantization algorithm design and GPU inference-kernel co-design, with a focus on reasoning LLMs that generate long chains of thought.
• Technical level: Advanced. The paper assumes familiarity with linear quantization, outlier channels, orthogonal transforms, Givens rotations, and CUDA kernel parallelism.
• Scope: The paper proposes ParoQuant, a 4-bit weight-only PTQ method built on "scaled pairwise rotation," and evaluates its accuracy and speed against AWQ, EfficientQAT, QTIP, QuIP#, OmniQuant, and SpinQuant.

What This Paper Is About

Quantizing LLM weights and activations to low precision shrinks memory use and speeds up inference, but outlier values in weights and activations make low-bit quantization lossy. Existing fixes either leave too much error or add heavy runtime computation, and the problem gets worse for reasoning models because errors accumulate across the tens of thousands of tokens generated in a long chain of thought. ParoQuant's goal is to suppress outliers well enough to match the best quantization accuracy while adding very little inference overhead.

Key Contributions

1. Scaled pairwise rotation transform. The paper introduces a transform combining independent Givens rotations (rotations applied only to selected channel pairs) with channel-wise scaling, applied to weights before quantization. Channel-wise scaling evens out magnitudes across channels; the pairwise rotations narrow the dynamic range within each quantization group.
2. Independence as a system design constraint. By requiring that pairs within a rotation be mutually independent (each channel appears in only one pair, formally defined as Independent Pairs and Independent Rotation), the method removes sequential dependencies, so all rotations for a pair set are fully parallelizable on GPUs. Independent rotations also align naturally with block-wise quantization because each channel group can get its own pair selection.
3. Layer-wise two-stage optimization. Stage 1 optimizes rotation angles and channel-wise scaling factors; Stage 2 uses a QAT-like procedure (similar to EfficientQAT) to fine-tune weights and the linear quantization parameters s and z. The layer loss uses the output of already-quantized preceding layers, letting later layers compensate for earlier quantization error.
4. A fused CUDA kernel co-designed with the transform. The kernel parallelizes at three levels: token, channel group, and pair. Because group size (e.g., 128) is small, activations fit in on-chip shared memory and rotation parameters fit in registers, and multiple independent rotations can be fused with a one-time memory load.

Main Findings

• Accuracy over AWQ on reasoning tasks: ParoQuant achieves an average 2.4% accuracy improvement over AWQ on reasoning tasks, with less than 10% overhead. In Section 5.2 it is described as only about 10% slower than AWQ.
• AWQ's degradation on reasoning: The accuracy of Qwen3-4B on MMLU-Pro drops sharply from 71.0 to 68.2 after 4-bit AWQ quantization — a 2.8% drop — which the authors attribute to quantization errors accumulating at each decoding step.
• Matches QTIP while being faster: ParoQuant matches the accuracy of QTIP while being about 25% faster; Section 5.2 reports being 15%–30% faster than QTIP. In the perplexity table, ParoQuant's geometric-mean speedup is 2.2× versus 1.7× for QTIP and 2.4× for AWQ.
• Reasoning benchmark averages: ParoQuant causes only an average 0.9% accuracy degradation and achieves 6.3%, 2.4%, and 0.9% improvements over EfficientQAT, AWQ, and QTIP respectively. On MMLU-Pro, ParoQuant scores 57.1 (R1-Distill-Llama-8B), 70.1 (Qwen3-4B), 74.1 (Qwen3-8B), and 77.5 (Qwen3-14B), against FP16 values of 58.8, 71.0, 74.6, and 78.1.
• Non-reasoning benchmarks: On commonsense tasks with thinking mode disabled, ParoQuant outperforms AWQ, EfficientQAT, and QTIP by 0.9%, 0.7%, and 0.2% respectively. The gap is smaller than on reasoning tasks because these benchmarks generate only a few tokens, so error accumulation is minimal.
• Perplexity: ParoQuant gives the best results among linear quantization methods across all tested sizes, and it outperforms QuIP# and matches QTIP even though linear quantization is inherently more error-prone than vector quantization. For LLaMA-3-8B on WikiText2: FP16 5.54, ParoQuant 5.73, QTIP 5.69, AWQ 5.92, EfficientQAT 5.87, QuIP# 5.81, SpinQuant 5.83.
• Throughput (tokens/s, batch size 1, RTX A6000): For Qwen3-1.7B, FP16 170, AWQ 320, QTIP 209, ParoQuant 278; for Qwen3-4B, FP16 78, AWQ 176, QTIP 117, ParoQuant 160; for LLaMA-3-8B, FP16 45, AWQ 120, QTIP 95, ParoQuant 112; for Qwen3-14B, FP16 25, AWQ 70, QTIP 55, ParoQuant 65.
• Sparse rotations are nearly as good as full ones: Optimizing only the top 10% channel pairs with the largest magnitude difference is almost as effective at reducing quantization-induced output error as optimizing all pairs. A full n×n orthogonal matrix decomposes into at most ½n(n−1) Givens rotations, while one independent rotation of dimension n accommodates only n/2 pairs, a fraction 1/(n−1) of the full parameter count — which motivates stacking a small number (e.g., 8) of independent rotations.
• Ablation on transform components: With LLaMA-3-8B (C4 perplexity), "8 independent rotations + scaling" gives 7.27 with Stage 2 and 7.35 without; no transform gives 7.56 without Stage 2 and 7.42 with it. Channel-wise scaling alone gives 7.40/7.41.
• Calibration efficiency: ParoQuant achieves strong performance with as few as 128 training samples (C4 7.30, MMLU 69.5). Accuracy improves as the number of rotations increases up to 8 (at 2048 samples: 0 rotations gives C4 7.41 / MMLU 69.6; 8 rotations gives C4 7.27 / MMLU 70.1). Using 2048 samples from RedPajama alone is slightly worse than the mixed dataset, indicating greater training-set diversity improves generalization.
• Kernel speedup trend: The speedup of the transform (with 8 independent rotations) over the fast Hadamard transform on an RTX A6000 increases with channel dimension, because the Hadamard transform has inherent dependencies across all channels.

Methodology in Plain English

The starting point is the standard trick of inserting an invertible transform T into a linear layer so that XW = (XT⁻¹)(TW), then quantizing TW instead of W. Two families of T exist: diagonal channel-wise scaling, which is cheap and can often be merged into neighboring operators, and rotations, which suppress outliers better but usually need online matrix multiplication in FP16.

The authors build their transform in three steps. First, they replace a dense orthogonal matrix with Givens rotations, each acting on just two rows of the weight matrix using a few vectorized multiply-and-add instructions — no matrix multiplication needed. Second, they force all the chosen channel pairs to be independent (each channel used at most once), which removes ordering dependencies and makes every rotation computable in parallel; this also means each 128-channel quantization group can have its own pair selection. Third, since a single independent rotation has far fewer tunable parameters than a full orthogonal matrix, they stack a small number of independent rotations (8 in the experiments) and multiply in per-channel scaling factors to handle magnitudes across the whole matrix and to catch isolated outliers.

Pairs are selected by shuffling all candidate pairs and greedily picking available ones, skipping pairs already used in earlier rotations to diversify combinations (Algorithm 1). Optimization proceeds layer by layer: optimize angles and scaling to minimize the layer's output error, then fine-tune weights and quantization parameters, always comparing against the original layer's output while feeding in the output of the already-quantized preceding layers (Algorithm 2). Inference is done by a single fused CUDA kernel that applies the inverse rotations and scaling to activations, parallelized over tokens, channel groups, and pairs.

Why This Matters

Impact on research. The paper reframes outlier handling as a parameter-efficiency question: if only a sparse subset of rotation pairs is needed, rotations become both optimizable and cheap, which weakens the assumed trade-off between quantization fidelity and inference speed. It also argues that reasoning workloads, where errors compound over long generations, are a distinct and harsher evaluation setting than short-answer benchmarks, and it provides accuracy-and-throughput evidence on models from 1.7B to 70B.

Real-world applications (from the paper's framing):

• Edge and on-device deployment of LLMs, where weight-only 4-bit quantization reduces memory footprint and increases throughput in memory-bound autoregressive decoding.
• Large-batch LLM serving, where quantizing activations as well enables lower-bitwidth matrix multiplications and cuts computational cost.
• Serving reasoning models that emit long chains of thought, where per-step quantization error would otherwise accumulate.
• General deployment pipelines built on existing inference frameworks, since the method targets linear quantization, which is more widely supported than vector quantization.

Industry relevance. ParoQuant is designed as a drop-in on top of the Transformers library, with only the weight transform and dequantization code modified, and the authors report speedups measured on an RTX A6000 at batch size 1. Its combination of near-AWQ speed with accuracy that matches QTIP addresses the practical selection problem practitioners face when choosing a 4-bit scheme.

Future Directions

• Extending beyond weight-only quantization. The paper states the method can be extended to weight-activation quantization (Section A.3) and shows it is applicable there, but the reported experiments center on 4-bit weight-only linear quantization; broader validation is left open.
• Extending to vector quantization. The authors note the same method can be generalized to vector quantization but concentrate on linear quantization because it is more efficient and better supported by existing frameworks.
• Rotation count and pair-selection policy. Accuracy improves as rotations increase up to 8, and pairs are selected by a shuffled greedy rule with a skipped-pair constraint that the authors say can leave some rotations with too few pairs. Better selection strategies, or more rotations, remain unexplored.
• Structured pruning of the remaining outliers. The paper observes that isolated outliers may survive rotations and scaling and are handled instead by Stage-2 fine-tuning, leaving room for transforms that address them directly.

Target Audience

Researchers and engineers working on LLM quantization and inference systems will benefit most: those implementing PTQ pipelines, those writing or optimizing low-bit CUDA kernels, and those evaluating whether a 4-bit reasoning model can be deployed without large accuracy loss. Readers need a working understanding of quantization arithmetic, orthogonal transforms, and GPU execution models to follow the method sections, though the motivation and results sections are accessible to a broader machine-learning audience.

Authors’ abstract

Post-training quantization (PTQ) compresses the weights and activations of large language models (LLMs) into low-precision representations to reduce memory footprint and accelerate inference. However, the presence of outliers in weights and activations often leads to large quantization errors and severe accuracy degradation, especially in recent reasoning LLMs where errors accumulate across long chains of thought. Existing PTQ methods either fail to sufficiently suppress outliers or introduce significant overhead during inference. In this paper, we propose Pairwise Rotation Quantization (ParoQuant), a PTQ method that combines hardware-efficient and optimizable independent Givens rotations with channel-wise scaling to even out the magnitudes across channels and narrow the dynamic range within each quantization group, effectively addressing the outlier issue. We further co-design the inference kernel to fully exploit GPU parallelism and keep the rotations and scaling lightweight at runtime. Under weight-only quantization, ParoQuant achieves an average 2.4% accuracy improvement over AWQ on reasoning tasks, with less than 10% overhead. ParoQuant also matches the accuracy of state-of-the-art weight-activation quantization methods. This paves the way for more efficient and accurate deployment of reasoning LLMs.