Fractional Power-of-Two Quantization for Efficient and Effective Multiplier-Free LLM Inference

AI summary

Overview

Research area: Efficient machine learning systems — post-training quantization (PTQ) of Large Language Models, co-designed with digital hardware arithmetic (shift-add datapaths).

Technical level: Advanced. The paper combines quantization theory (Gaussian-proxy grid analysis, entropy-based metrics), LLM PTQ pipelines (OSTQuant, GPTQ), and gate-level / RTL hardware cost modeling.

Scope: A single paper proposing FPoT, a base-√2 "half-octave" logarithmic weight quantization grid that enables multiplier-free 4-bit LLM inference with accuracy close to uniform INT4 baselines.

What This Paper Is About

Most of the compute and energy in LLM inference goes to multiply-accumulate (MAC) operations inside Transformer linear projections. Power-of-Two (PoT) weight quantization replaces those multipliers with simple bit-shifts, but the standard base-2 grid is so coarse that it badly damages LLM accuracy at 4 bits. This paper asks whether a slightly finer logarithmic grid — one with two levels per octave instead of one — can keep the multiplier-free hardware benefit while recovering the accuracy, and answers yes, provided the hardware approximation of √2 is folded into the quantization process itself.

Key Contributions

1. A theoretically grounded base-√2 grid for 4-bit LLMs. Using a Gaussian proxy for preprocessed weight distributions, the authors identify k = 2 (half-octave, levels spaced by 2^(1/2) ≈ 1.414) as the best integer-resolution PoT grid at 4 bits, cutting worst-case relative error from 41.4% to 18.9% — roughly a 2.2× improvement in grid resolution.

2. Approximate-grid quantization. Rather than quantizing to an exact √2 grid and then swapping in an approximate hardware coefficient, FPoT quantizes weights directly to the actual hardware grid (using 1.5 for odd exponents, realized as x + x≫1 with one adder). Weight calibration absorbs the 6.1% coefficient error as part of the quantization grid instead of as post-hoc distortion.

3. A calibration-time grid-alignment refinement. A phase-shift search over G candidate scales s_g = s_base · 2^(g/G), picking per output-channel the candidate minimizing MSE. Because weights are static after calibration, the offset is absorbed into the per-channel scale at zero inference overhead.

4. A hardware and accuracy evaluation across seven models and three bit-widths. FPoT is paired with uniform INT activations and KV cache and composed with state-of-the-art PTQ preprocessing (OSTQuant, with DuQuant portability checked), yielding a processing element (PE) with roughly 0.50× the compute-unit cost of a 4-bit integer multiplier.

Main Findings

• Large accuracy recovery over standard PoT. On Llama-3-8B at W4A4KV4, FPoT reaches 7.60 PPL and 64.8% 0-shot-9 accuracy versus PoT's 8.79 PPL / 59.6% and the uniform INT baseline's 7.36 PPL / 65.2%. PPL overhead drops from +1.43 (PoT) to +0.24 (an 83% reduction), and downstream accuracy degradation shrinks from −5.6%p to −0.4%p.

• An 8.72% relative zero-shot gain over PoT. The abstract and introduction report FPoT improving zero-shot commonsense reasoning by 8.72% relative to the PoT baseline on Llama-3-8B at 4-bit weights and activations, while staying within 0.4%p of the uniform integer baseline.

• Consistency across 21 configurations. FPoT is evaluated on seven models (Llama-3-8B, Llama-2-7B, Llama-2-13B, Qwen2.5-1.5B, Qwen2.5-7B, Qwen2.5-14B, Mistral-7B) and three settings (W4A4KV4, W4A8KV8, W8A8KV8). Across the 14 W4 configurations, ΔPPL ranges from +0.05 (Qwen2.5-14B W4A4KV4) to +0.26 (Qwen2.5-1.5B W4A8KV8), strictly below +0.3; PoT overhead reductions are 71%–94% (67%–94% across all 21 configurations). Zero-shot accuracy lands within 1.0%p of uniform INT in 20/21 configurations (mean gap −0.17%p), and all 14 W4 configurations are within 0.5%p.

• k = 2 is close to the analytical optimum. Minimizing MSE on N(0,1) weights with 15 levels and a jointly optimized scale gives β* ≈ 2^0.55 (effective k* ≈ 1.8). At k = 2, MSE is 1.27× that of INT4, versus 3.21× for PoT (k = 1), 2.87× at k = 3, and 5.37× at k = 4 (dynamic-range starvation). The optimum is bit-width specific: at 8 bits it shifts to k* ≈ 19.

• A new metric favors the half-octave grid. The authors introduce Normalized Bin Entropy (NBE), the fraction of the fixed log2 N bit budget that carries information. Under the Gaussian proxy, FPoT k=2 scores NBE = 0.9364, above uniform INT4 (0.9026) and PoT (0.8773).

• Implicit magnitude pruning. The k-adaptive zero threshold τ_k = 2^(−1/(2k)) equals ≈0.841 at k = 2, zeroing about 22% of preprocessed weights. Those MACs are skipped entirely via zero-gating, and the loss is compensated by the 2× finer grid.

• Approximate-grid quantization beats both alternatives. On Llama-3-8B W4A4KV4, naive post-hoc substitution of the 1-adder 1.5× coefficient costs +0.17 PPL (7.729), but approximate-grid quantization reaches 7.574 — within +0.01 of the exact-√2 result (7.560). The ordering is Approx Grid > Aware training (7.625) > Exact Grid (7.729) for the 1-adder case.

• The 1-adder sits at the Pareto knee. A 3-adder variant already costs 1.00× INT4 compute units; the 2-adder costs 0.75× and the 1-adder 0.50×; PoT's 0-adder costs 0.38× but incurs an unusable +1.19 PPL penalty. The 2-adder and 3-adder options pay 1.5× and 2× more compute units for at most a 0.005 PPL improvement.

• Energy estimates. Anchored to RTL synthesis on Samsung 5 nm (250 MHz, TT corner, 0.6 V, 25°C, 50% toggle rate) reporting a PoT-to-INT ratio of 0.42× at W4A8, FPoT is reported at 0.42–0.46× at W4A4 (0.84–1.00× at W4A8; 1.68–2.00× at W8A8) — a 2.2–2.4× energy reduction per weight multiply versus INT, with the √2 unit contributing at most 0.04× overhead.

• Preprocessing is essential. Without any transform, FPoT yields a catastrophic PPL of 181.3; only the full combination of learned rotation plus per-channel scale reaches 7.56, within +0.12 of the non-PoT baseline.

• The logarithmic grid does not transfer to activations. Applying FPoT to both weights and activations gives PPL 16.9–114.1 at k=1 and 8.21–52.49 at k=2, versus 7.56 for weight-only, so uniform INT activations remain the practical choice.

• Robustness and portability. Calibration is stable (|ΔPPL| ≤ 0.065 across n ∈ {128, 256, 512}; +0.06 to +0.10 shift from WikiText-2 to C4), zero-threshold and rounding-mode choices matter little (ΔPPL < 0.03), a different PTQ framework (DuQuant) gives +0.05 to +0.34 PPL overhead, and LogART degrades catastrophically (PPL > 3900) under post-hoc INT4 activation where FPoT stays stable.

Methodology in Plain English

The authors start from a hardware observation: if every weight is constrained to a signed power of two, a multiply becomes a bit-shift plus an add. The problem is that powers of two are spaced far apart (each level is twice the previous one), so rounding a weight to the nearest one can be off by up to 41.4%.

Their fix is to allow half-steps in the exponent — levels at 2^(e/2) rather than 2^e. This doubles the number of levels the same 4-bit code can address and halves the worst-case error, but it introduces a √2 factor for odd exponents. They implement √2 approximately as 1.5× (one add and one shift: x + x≫1), which is 6.1% off. The key trick is that they do not quantize to the true √2 grid and patch the hardware afterwards — they quantize to the grid the hardware actually implements. Because weight calibration happens after this choice, the 6.1% error is absorbed into the choice of quantization levels rather than appearing as a distortion at inference.

Around this core they add a cheap refinement: after calibration, they try G slightly different per-channel scales (offsets of 2^(g/G)) and keep whichever minimizes reconstruction error. Since the scale is a static constant baked into the deployed model, this costs nothing at runtime.

To justify the choice of "half-octave," they argue that after state-of-the-art PTQ preprocessing redistributes outlier channels, the log-magnitude of weights looks roughly Gaussian, with a spread of about 1.02 octaves and ~95% of mass within ~4 octaves. Under that model they can compute MSE and their new entropy metric numerically for different grid shapes, confirming that two levels per octave is near the sweet spot at 4 bits and that one level per octave wastes levels out in the sparse tail.

Experimentally, they bolt this weight grid onto the OSTQuant preprocessing pipeline (learned rotations plus per-channel smooth scales, optimized for 400 steps), finish with GPTQ calibration on the PoT target grid, and evaluate WikiText-2 perplexity plus the mean of nine zero-shot benchmarks via lm-eval-harness. To fit 14B-scale models on a single A100-80GB GPU, they write a fused Triton kernel merging quantize–dequantize–STE into one launch, cutting per-projection intermediate memory by 82%. Hardware cost is estimated with a simple component model (multipliers scale as W×A gates; shifters as W+A) cross-checked against Yosys logic synthesis and anchored to published Samsung 5 nm RTL data.

Why This Matters

Impact on research. The paper closes a gap between two previously separate lines of work: activation-quantization methods that enable fully integer arithmetic but retain multipliers, and PoT methods that remove multipliers but either target CNNs or keep FP16 activations. FPoT is presented as the only multiplier-free PTQ reported to achieve near-lossless W4A4KV4. The NBE metric and the Gaussian-proxy grid analysis are also reusable design tools for choosing quantization grids rather than tuning them by trial and error.

Real-world applications.

• On-device and edge LLM inference, where arithmetic energy and gate area are the binding constraints rather than memory bandwidth.
• Prefill-heavy datacenter serving, which the paper identifies as the compute-bound regime where weight-MAC energy dominates and the gains are largest.
• Custom NPU/ASIC accelerator design, where replacing every 4-bit integer multiplier in every linear projection with a shifter, one adder, and a multiplexer reduces silicon area.
• Battery-powered or thermally constrained devices running small models such as Qwen2.5-1.5B, one of the seven models evaluated.

Industry relevance. The work is co-authored with LG Electronics and anchors its energy claims to Samsung 5 nm RTL synthesis, and the authors frame it explicitly as a pathway toward shift-based LLM inference hardware. Because FPoT composes with existing PTQ frameworks (OSTQuant as the main pipeline, DuQuant validated for portability) rather than replacing them, it is a drop-in weight-grid substitution rather than a new training regime.

Future Directions

• End-to-end RTL/ASIC validation. The efficiency projections currently rest on analytical cost models anchored to published RTL data, with Yosys synthesis confirming only the structural ordering (INT > FPoT > PoT) and bit-width scaling. Full synthesis and silicon measurement are left for future work.
• Scaling beyond 14B parameters. Preprocessing training requires float32 weight copies, which limits single-GPU (80 GB) experiments to ≤14B parameters. Whether the accuracy conclusions hold at larger scale is open.
• Bit-width generalization. The k = 2 optimum is specific to 4 bits; at 8 bits the analysis points to k* ≈ 19. A systematic mapping from bit-width to grid resolution — and evaluation at those other widths — remains to be done.
• Compatibility with other calibration methods. All results combine state-of-the-art PTQ preprocessing with GPTQ; compatibility with AWQ or plain round-to-nearest is untested. Extending the logarithmic grid to the KV cache, which currently stays uniform INT, is also untouched.

Target Audience

Researchers and engineers working on LLM quantization, efficient inference, and ML hardware accelerators — particularly those interested in logarithmic or shift-based arithmetic and in co-designing quantization grids with the datapath that executes them. Readers need familiarity with post-training quantization, GPTQ

Authors’ abstract

Large Language Models (LLMs) demand efficient inference, where the multiply-accumulate (MAC) operations in linear projections dominate compute and energy cost. Post-training quantization (PTQ) reduces this cost by mapping weights and activations to low-bit integers. Power-of-Two (PoT) weight quantization promises efficient multiplier-free LLM inference via shift-add logic, but suffers significant accuracy degradation at low bitwidths due to its coarse base-2 quantization grid. We introduce Fractional PoT (FPoT), a base-$\sqrt{2}$ (half-octave) fractional PoT grid that achieves practical 4-bit weight quantization while preserving a fully integer shift-add datapath. The method combines (i) the base-$\sqrt{2}$ weight grid with uniform integer activations, (ii) a calibration-time grid-alignment refinement absorbed into the static per-channel scale, and (iii) a lightweight 1-adder $\sqrt{2}$ approximation whose error is absorbed during weight calibration via approximate-grid quantization, yielding a processing element (PE) with significantly fewer gates than a 4-bit integer multiplier. On Llama-3-8B with 4-bit weight and activation quantization, FPoT improves zero-shot commonsense reasoning performance by 8.72% relative to the PoT baseline, while maintaining performance close to the uniform integer baseline, with only a negligible 0.4%p degradation. These results hold across various configurations spanning seven models and three bit-widths. Furthermore, FPoT can be integrated with state-of-the-art LLM PTQ methods, demonstrating framework-agnostic applicability.