HyQuant: Hybrid-Precision Quantization for LLM Attention

AI summary

Overview

Research area: Efficient large language model (LLM) inference, specifically quantization of the attention module and KV cache for long-context, chain-of-thought workloads.

Technical level: Intermediate. Readers should be comfortable with attention, KV caches, Prefill vs. Decode stages, and basic quantization concepts (bit-widths, dequantization, MSE error analysis).

Scope: The paper proposes and evaluates HyQuant, a hybrid-precision quantization framework that keeps a tiny set of "vertical-line" tokens and a local sliding window in full precision while quantizing the rest of attention to low bits, for both Prefill and Decode.

What This Paper Is About

Low-bit quantization of the attention module in LLMs saves compute and memory but introduces large errors at very low bit-widths, degrading model quality. HyQuant's goal is to allocate precision non-uniformly: it identifies a small set of persistently important "vertical-line" key positions plus a recent local window and keeps those in full precision, while quantizing the remaining majority to low bits. The framework covers both the compute-bound Prefill stage and the memory-bound Decode stage in a single integrated design.

Key Contributions

1. Vertical-line-aware precision allocation. HyQuant identifies a small set of persistently attended key positions (typically fewer than 5% of tokens) using accumulated column-wise attention mass, and retains them in full precision rather than treating all tokens uniformly.

2. A hybrid-precision Prefill attention operator. Keys and values are split into a low-bit quantizable set and a full-precision set (vertical-line positions plus a local window), then processed in a single fused, FlashAttention-style online-softmax kernel with segmented scanning.

3. A hybrid low-bit KV cache for Decode with fused dequantization. The same vertical-line and window positions stay in full precision in the cache, while the rest is stored at low bit-width; dequantization is fused into the attention kernel so full-precision intermediates are never materialized.

4. A departure from prior pattern-based methods. Where MInference uses vertical-line structures as a sparsity mask, HyQuant uses them to assign different precisions to tokens, retaining information from less significant tokens in low-bit form. The paper also notes that KVTuner's sensitivity-aware mixed precision is applied to layers rather than tokens, and that HyQuant addresses both Prefill and Decode rather than one stage.

Main Findings

• Concentrated attention mass: On Llama-3.1-8B, the global top-1% key positions cover 58.83% of attention mass, top-5% covers 64.09%, and top-5% plus a local window of W=128 covers 85.63%. On Qwen3-8B the corresponding figures are 47.30%, 54.10%, and 82.53%.

• Errors concentrate at high-score positions: Error analysis on Qwen3-8B (Layer 28, with full-precision FlashAttention as reference) shows uniform 4-bit quantization yields noticeably larger MSE than uniform 8-bit. When the rest is 4-bit, keeping only the top-1% or top-5% high-score positions in full precision dramatically suppresses error and consistently approaches the 8-bit error level across sequence lengths from 1K to 32K.

• LongBench v1 accuracy: Average scores with HyQuant (K4V4, top-5%) were 45.04 on Qwen3-8B versus 44.59 for FA2 full precision, 37.68 for KIVI, 38.13 for SageAttention, and 40.45 for KVTuner; 46.73 on Llama-3.1-8B-Instruct versus 46.63 for FA2, 45.68 for SageAttention, 45.66 for KVTuner; 45.78 on GLM-4-9B-0414 versus 44.83 for FA2, 45.75 for SageAttention, 45.63 for KVTuner; and 48.46 on Qwen3-32B versus 48.61 for FA2, 41.98 for KIVI, 48.22 for KVTuner. The authors state that HyQuant slightly exceeding FA2 on some datasets is normal evaluation variance, not evidence that quantization improves capability. Results on GSM8K and MATH500 are named as evaluation benchmarks but their specific scores are not reported in the available content.

• Decode kernel latency: Against FA2, HyQuant reached speedups of 1.32x at a 1,024-token prefix (0.224 vs. 0.170 ms/token), 2.40x at 2,048 (0.416 vs. 0.173), 3.06x at 4,096 (0.805 vs. 0.263), 3.36x at 8,192 (1.604 vs. 0.478), 3.52x at 16,384 (3.181 vs. 0.903), and 3.58x at 32,768 (6.354 vs. 1.775).

• End-to-end decode speedup is more moderate: Normalized to FA2, HyQuant reaches 1.04x at 1,024, 1.07x at 2,048, 1.09x at 4,096, 1.12x at 8,192, 1.13x at 16,384, and 1.17x at 32,768. KIVI and KVTuner stay below 1.0x across these lengths (0.72x to 0.69x for KIVI; 0.78x to 0.72x for KVTuner).

• Prefill operator-level error: Prefill latency is described as comparable to SageAttention, so the paper reports numerical deviation instead of a Prefill speedup claim. Using FA2 as reference, HyQuant achieves a large reduction in layer-wise MSE over SageAttention, reported as the factor MSE_Sage / MSE_HyQuant, with the first five layers visualized.

• Throughput under high parallelism (Qwen3-8B, 32K prefix, single H100-80GB): At batch 1, FA2 42.8, HyQuant 43.7, KIVI 23.7, KVTuner 24.2 tokens/s. At batch 4: 122.2, 84.2, 32.5, 33.4. At batch 8: 152.7, 155.6, 34.9, 35.6. At batch 16, FA2, KIVI, and KVTuner are OOM while HyQuant runs at 231.6 tokens/s. At batch 32 all methods are OOM. The authors attribute this to the fused "dequantize-in-attention" design versus the "dequantize-then-attend" strategy of their KIVI and KVTuner implementations.

• Ablations: Both the local full-precision window and vertical-line-aware retention reduce Prefill-stage MSE on Qwen3-8B with LongBench v1. Increasing the retained vertical-line token ratio generally improves accuracy and reduces quantization error but raises the full-precision KV budget; a larger window size slightly improves accuracy. Top-5% is used as the practical trade-off.

• Overhead: Vertical-line identification accounts for only 3% to 5% of total runtime. HyQuant buffers at most 64 query vectors in FP16, costing 64 × H_Q × d × 2 bytes. Keeping 5% of vertical-line tokens in full precision increases the non-window KV cache size by about 15% compared with strict 4-bit quantization, with total overhead also depending on local-window size.

Methodology in Plain English

The researchers started from an empirical observation: in attention heatmaps across model families, a few key positions form persistent bright vertical columns that many queries attend to, while the rest of the sequence receives much less attention. Instead of compressing every token equally, they split keys into three disjoint groups per layer and head: vertical-line positions, a fixed-size recent sliding window, and the quantized remainder.

To find the vertical-line positions cheaply, they average a set of recent query vectors, score every non-window key position against that proxy, and take the top fraction (ρ) by score. For grouped-query attention models the scoring is done in a GQA-native form without materializing repeated KV heads. This adds only a lightweight reduction, reported at 3% to 5% of runtime.

For Prefill, the key/value sequence is partitioned into a low-bit quantizable prefix, a full-precision vertical-line region, and a full-precision local window. Tiles from all three regions are scanned in one segmented FlashAttention-style online softmax pass, with scales of c_B = 127 for INT8 and c_B = 448 for FP8 used in quantization. For Decode, the KV cache itself is stored hybrid: full precision for vertical-line and window positions, low bits for the rest. The decode kernel dequantizes blocks on the fly inside the same online-softmax pass rather than materializing a full-precision cache, reducing memory traffic.

Experiments were run on NVIDIA H100 GPUs. Default settings retain the online-identified top-5% vertical-line tokens and a local sliding window, with all remaining KV positions stored as Key-4bit, Value-4bit. Accuracy was measured on LongBench v1, GSM8K, and MATH500 across Qwen3-8B, Qwen3-32B, Llama-3.1-8B-Instruct, and GLM-4-9B-0414, against FA2, KIVI, KVTuner, and SageAttention baselines.

Why This Matters

Impact on research: The work reframes attention quantization as a non-uniform precision-allocation problem driven by measured attention structure, rather than a uniform bit-width or outlier-smoothing problem. It also argues against treating Prefill and Decode separately, showing a single token-importance signal can drive both an operator design and a cache design. The authors report code at https://github.com/jerrysfls/HyQuant.

Real-world applications:

• Long-context chain-of-thought reasoning services, where multi-step traces reach tens of thousands of tokens and Decode memory bandwidth dominates cost.
• Batch-serving deployments on single-GPU nodes, where the reported ability to run at batch size 16 (when FA2, KIVI, and KVTuner are OOM at a 32K prefix) directly increases tokens served per device.
• Retrieval and document-aggregation workloads over long inputs, represented in the evaluation by Single-Document QA, Multi-Document QA, Summarization, Few-shot Learning, and Semantic tasks in LongBench v1.
• Mathematical reasoning pipelines, evaluated on GSM8K and MATH500.

Industry relevance: The reported end-to-end decode speedups of 1.04x to 1.17x and the 15% KV cache size increase versus strict 4-bit describe a concrete accuracy-versus-memory trade-off that serving teams can weigh. The 3% to 5% runtime overhead for vertical-line identification and the use of Triton and existing low-precision backends (INT8/FP8, INT4/FP4) suggest the approach is intended to be deployable without exotic hardware support.

Future Directions

• Short-context behavior: The authors state that the benefit of retaining vertical-line tokens is most visible in long-context tasks, and that HyQuant may not deliver the same level of improvement on short-context tasks, though it remains competitive with the full-precision baseline (Appendix B.4, Table 13).
• Hardware dependence of gains: Because experiments use NVIDIA H100 GPUs, the paper notes that end-to-end speedups can be less pronounced at short context lengths on high-end GPUs where the decoding pipeline is not fully memory-bound.
• Scaling beyond the tested models: GPU memory limits prevented evaluation of models larger than Qwen3-32B, such as Qwen3-80B.
• Agentic and coding settings: The authors state it is unknown whether the method remains effective in agent or coding-related settings.

Target Audience

Researchers and engineers working on efficient LLM inference, KV-cache compression, and long-context serving will benefit most. It is also relevant to practitioners choosing between quantization schemes such as KIVI, KVTuner, and SageAttention, and to systems researchers interested in fused attention kernels that mix precisions within a single operator. Readers without background in attention mechanics and quantization would need to build that context first.

Authors’ abstract

Quantization has been widely adopted in LLM training and inference to reduce cost and improve efficiency. However, low-bit quantization of the \emph{attention} module often introduces large errors at very low bit-widths, causing performance degradation. Existing methods mainly rely on smoothing techniques to handle outliers, while we propose a hybrid quantization design to better balance accuracy and efficiency. Specifically, we propose \textbf{HyQuant}, an efficient hybrid quantization framework for LLM attention. HyQuant quantizes most attention states into low-bit formats while retaining a small set of vertical-line tokens and local-window states in high precision. These accuracy-critical regions are selected using lightweight vertical-line-aware attention-pattern signals, reducing quantization error with limited overhead. In the Prefill stage, HyQuant uses a hybrid-precision quantized attention operator that preserves vertical-line tokens and a local sliding window in full precision while quantizing the remaining context. In the Decode stage, HyQuant applies the same principle to KV-cache compression and fuses KV dequantization with attention computation to improve memory and hardware efficiency. Across diverse tasks, models, and datasets, HyQuant maintains nearly lossless accuracy with an extremely simple design, demonstrating the efficiency and practical feasibility of hybrid quantization for LLM attention. Code is available at: https://github.com/jerrysfls/HyQuant .