TurboBoA: Faster and Exact Attention-aware Quantization without Backpropagation

AI summary

Overview

Research area: Post-training quantization (PTQ) for large language models (LLMs), specifically backpropagation-free, Hessian-guided weight quantization.

Technical level: Advanced. The paper assumes familiarity with GPTQ-style error compensation, Kronecker-structured Hessians, Cholesky decomposition, and attention reconstruction losses.

Scope: The paper introduces TurboBoA, a backpropagation-free PTQ algorithm that accelerates the attention-aware BoA method by jointly quantizing multiple out-channels with a closed-form error compensation rule, and adds cross-layer error correction plus adaptive grid refinement.

What This Paper Is About

Quantizing LLMs to low bit-widths after training (PTQ) is essential for cutting memory and compute costs, but the widely used GPTQ assumes layers are independent, which causes severe accuracy loss at low bit-widths such as INT2. BoA fixed this by baking inter-layer dependencies inside attention modules into the Hessian, but it must quantize out-channels strictly sequentially, making it far slower than GPTQ. TurboBoA aims to keep BoA's accuracy advantage while removing its sequential bottleneck.

Key Contributions

1. Joint quantization of multiple out-channels (F1). TurboBoA quantizes N out-channels simultaneously instead of one at a time, with a closed-form error compensation rule (Proposition 3.1) that explicitly encodes the dependencies between the jointly quantized channels. In the paper's example, a 128-out-channel weight matrix drops from 128 sequential operations to 8 when N = 16, and timing measurements show more than a three-fold speedup over BoA.

2. Error compensation for pre-quantized layers (F2). The method reformulates the compensation problem to account for input deviation caused by quantization errors in earlier layers, and provides a closed-form solution (Proposition 3.2) for general dense out-channel Hessians — unlike GPTAQ, which assumes the out-channel Hessian is the identity matrix.

3. Adaptive grid computation with coordinate descent refinement (F3). Quantization grids are recomputed immediately before each out-channel is quantized so they stay aligned with updated weights, and scales are then refined by coordinate descent under frozen integer weights (Proposition 3.3) to reduce attention reconstruction error.

4. Empirical validation. Experiments across Llama2 and Llama3 models show TurboBoA is substantially faster than BoA while improving accuracy, reaching state-of-the-art results in both weight-only and weight-activation quantization when combined with outlier suppression techniques.

Main Findings

• More than three-fold speedup from joint quantization. In Table 2 (INT2, with QuaRot applied), BoA takes 13.32 minutes on Llama3.2-1B and 94.75 minutes on Llama3-8B, while TurboBoA with N = 16 takes 4.363 minutes and 25.30 minutes respectively. For the Llama3.1-70B model, the reduction is from 16.99 hours to 5.636 hours, which the paper describes as saving 9~12 hours in absolute terms.

• Speedup gains saturate beyond N = 16. Going from N = 4 to N = 16 cuts time significantly, but larger values (N = 32 or 64) yield only marginal gains, so N = 16 is used for all main experiments. The paper notes that TurboBoA reduces to GPTQ at N = 64 for Llama3.2-1B and N = 128 for other models.

• Accuracy degradation from joint quantization is negligible. Although increasing N reduces the number of out-channels available for error compensation (for example, from 127 to 112 when N = 16 in the 128-out-channel example), the paper reports that performance degradation remains negligible up to N = 64.

• Both enhancement features improve accuracy individually and combine additively. In Table 3 (INT2, Llama3.2-1B), starting from TurboBoA at 41.85 Wiki2 / 108.1 C4, adding F2 gives 37.15 / 92.58, adding F3 gives 39.45 / 107.3, and adding both gives 33.33 / 85.55 — versus BoA at 40.40 / 104.9. For Llama3-8B the combined setting gives 13.54 / 32.99 versus BoA at 15.20 / 36.95.

• Runtime overhead of the added features is small. F3 adds only a marginal cost (approximately one minute for Llama3-8B). F2 requires an additional forward pass of the full-precision model to compute the input deviation, but this is a fixed, one-time cost because the full-precision activation is independent of the quantization process.

• Weight-only quantization results. In Table 4(a) at INT2, TurboBoA outperforms BoA on all five reported models: Llama3.2-1B (33.33 vs. 40.86 Wiki2; 85.55 vs. 107.9 C4), Llama3.2-3B (24.10 vs. 33.40; 54.20 vs. 79.21), Llama3-8B (13.54 vs. 15.24; 32.99 vs. 36.82), Llama2-7B (9.108 vs. 10.42; 16.64 vs. 19.17), and Llama2-13B (7.337 vs. 8.237; 13.04 vs. 14.66). The FP16 baselines are 13.16/21.31, 11.05/16.49, 6.139/9.444, 5.473/7.266, and 4.885/6.730.

• Zero-shot accuracy gains. In Table 4(b) at INT2, TurboBoA reaches 40.31, 45.85, 52.59, 53.27, and 59.69 across the five models, versus BoA at 38.67, 43.86, 50.29, 51.00, and 56.92, against FP16 baselines of 56.82, 63.01, 70.34, 67.28, and 69.83.

• State-of-the-art in weight-activation quantization. In Table 5, under W2A4KV4 with the OSTQuant transform, TurboBoA reaches 46.10/111.7 (Llama3.2-1B), 24.53/72.72 (Llama3.2-3B), 14.51/38.12 (Llama3-8B), 9.142/16.59 (Llama2-7B), and 7.508/13.25 (Llama2-13B) on Wiki2/C4, improving over BoA in every case. The paper states that combined with outlier suppression techniques, TurboBoA achieves state-of-the-art results in both weight-only and weight-activation quantization.

• Experimental setup. Calibration used 128 sequences of length 2048 randomly sampled from WikiText-2. Metrics were perplexity on Wiki2 and C4 test sets plus average accuracy across eight zero-shot commonsense reasoning tasks (the paper's footnote lists ARC-challenge/easy, BoolQ, HellaSwag, LAMBADA, OpenbookQA, PIQA, and WinoGrande). All experiments ran on NVIDIA H100 GPUs (80 GB), with two GPUs used for the 70B model. The coordinate descent iterations were set to one, and the stabilization coefficient alpha was drawn from {0.05, 0.125, 0.25} with the best result reported per model.

Methodology in Plain English

The starting point is BoA's insight that quantization error in an attention layer can be measured more accurately by looking at the attention output rather than just the individual layer's reconstruction error. That produces a Hessian with two parts — one describing input correlations and one describing output-channel correlations — that can be factored apart (a Kronecker structure).

BoA's problem is that this richer Hessian forces a strictly sequential procedure: quantize one out-channel, compensate its error using all later out-channels, then move on. TurboBoA instead groups N out-channels and quantizes them together, then derives a closed-form formula (Proposition 3.1) that updates all remaining out-channels to absorb the combined error of the group. This trades some compensation flexibility for a large reduction in sequential steps.

Two further refinements are layered on top. First, because errors from earlier quantized layers distort the inputs seen by later layers, the compensation formula is extended (Proposition 3.2) to subtract an extra term involving the input deviation, so the quantized model tracks the full-precision model across depth. Second, instead of fixing the quantization grid once and reusing it, TurboBoA recomputes the grid just before each group is quantized, then freezes the integer weights and refines only the per-channel scales via coordinate descent (Proposition 3.3) to reduce the attention reconstruction error. A stabilization coefficient alpha, following GPTAQ, dampens the input-deviation correction to avoid numerical instability.

Why This Matters

Impact on research. The paper shows that the accuracy-versus-efficiency trade-off BoA introduced is not fundamental. Its propositions are deliberately written for any Kronecker-structured Hessian, not just BoA's, so the joint-quantization and correction machinery can be reused as better Hessian formulations appear. The paper also leaves the error-bound behavior as an explicit open question, giving theory-oriented work a target.

Real-world applications (implied by the paper's framing of PTQ for resource-constrained deployment):

• Serving large open-weight LLMs at reduced precision to cut GPU memory and inference cost in cloud deployments.
• Deploying quantized models on edge or on-device hardware where memory and compute are tightly limited.
• Enabling very low bit-width regimes (INT2, and W2A4 weight-activation) that would otherwise be unusable due to accuracy collapse.
• Compressing models in pipelines that already use outlier suppression such as SmoothQuant, QuaRot, SpinQuant, DuQuant, or OSTQuant, since the method is designed to combine with them.

Industry relevance. The work comes from Samsung Research, and the speedup matters operationally: the paper reports that for a 70B model, the reduction over BoA saves 9~12 hours in absolute terms per quantization run, and the code is slated for release at a public repository. Faster, more accurate PTQ directly affects how quickly production teams can iterate on quantized model releases.

Future Directions

• Formal error bounds as a function of N. The paper explicitly states that a formal theoretical characterization of the error bounds with respect to N is left as an interesting open question.
• Adopting stronger Hessians. Because Propositions 3.1–3.3 hold for any Kronecker-structured Hessian, the method can directly leverage more advanced Hessian formulations once they become available.
• Pushing joint quantization further. Since degradation stays negligible up to N = 64 but speedup gains flatten beyond N = 16, there is room to investigate whether a different compensation rule could sustain gains at larger N.
• Extended grid refinement. The paper fixes coordinate descent at one iteration because additional iterations yield only marginal improvements; whether other refinement schemes or more iterations change the accuracy-efficiency balance is not reported.

Target Audience

This paper is most useful to quantization researchers and ML systems engineers who already understand GPTQ-style Hessian-guided PTQ. Practitioners deploying Llama-family models at INT2, INT3, or W2A4 in production will benefit from the reported accuracy tables and runtime numbers, while researchers working on attention-aware or cross-layer PTQ will find the three propositions the most directly reusable component. Readers new to PTQ should treat the Kronecker Hessian derivations and the appendices as required background.

Authors’ abstract

The rapid growth of large language models (LLMs) has heightened the importance of post-training quantization (PTQ) for reducing memory and computation costs. Among PTQ methods, GPTQ has gained significant attention for its efficiency, enabling billion-scale LLMs to be quantized within a few GPU hours. However, GPTQ's assumption of layer-wise independence leads to severe accuracy drops in low-bit regimes. Recently, BoA improved upon GPTQ by incorporating inter-layer dependencies within attention modules, but its reliance on sequential quantization across all out-channels makes it substantially less efficient. In this paper, we propose TurboBoA, a new backpropagation-free PTQ algorithm that preserves the accuracy benefits of BoA while significantly accelerating the process. The proposed TurboBoA introduces three key innovations: (i) joint quantization of multiple out-channels with a closed-form error compensation rule, which reduces sequential bottlenecks and yields more than a three-fold speedup; (ii) a correction mechanism for errors propagated from preceding quantized layers; and (iii) adaptive grid computation with coordinate descent refinement to maintain alignment during iterative updates. Extensive experiments demonstrate that TurboBoA delivers substantial acceleration over BoA while consistently improving accuracy. When combined with outlier suppression techniques, it achieves state-of-the-art results in both weight-only and weight-activation quantization. The code will be available at https://github.com/SamsungLabs/TurboBoA.