AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMs

AI summary

Overview

Research area: Model compression and efficient inference for large language models, specifically post-training weight-only quantization, multi-precision model design, and GPU kernel co-design.

Technical level: Intermediate. Readers will get the most from this paper with some familiarity with quantization basics (bit-widths, scales, calibration), but the writing is largely self-contained.

Scope: This single paper presents AnyBCQ, a Binary-Coded Quantization (BCQ)-based framework that lets one quantized LLM run at 2-, 3-, or 4-bit precision, together with a matching CUDA kernel that executes directly on binary bit-planes.

What This Paper Is About

Large language models are expensive to deploy because of memory and latency bottlenecks, so practitioners quantize weights down to fewer bits. Real services have different latency, throughput, and accuracy budgets at different times, which motivates "multi-precision" models where a single stored model can be run at several bit-widths. The best existing multi-precision method, Any-Precision LLM, relies on clustering-based non-uniform quantization, which needs centroid table lookups and bit transposition at inference and degrades badly at 2 bits. AnyBCQ's goal is to build multi-precision capability on top of BCQ instead, so that inference can operate directly on binary bit-planes without those overheads, while keeping accuracy usable in the extremely low-bit regime.

Key Contributions

1. AnyBCQ, a BCQ-based multi-precision framework that shares binary bit-planes across precision levels while learning a separate set of scaling factors per precision, achieving strong low-bit accuracy and monotone improvement as more bits are enabled.

2. A progressive precision expansion mechanism in which binary codes from earlier precision levels are frozen, each new level adds a residual-derived bit-plane, and only the scaling factors are re-optimized.

3. A co-designed hardware-friendly CUDA kernel that exploits the binary basis representation, supports direct bit-plane-level operations, and enables per-request precision selection with negligible overhead.

4. An empirical demonstration of state-of-the-art accuracy-latency trade-offs across LLM benchmarks, plus a mixed-precision decoding case study, showing that one deployable model can serve diverse service-level objectives.

Main Findings

• Memory reduction: On Llama-3.1-8B with group size g=128, the proposed approach stores 1.09 GB of scales and 3.89 GB of binary terms for a total of 4.99 GB, compared to 9.85 GB total for a multi-model baseline that keeps a separate model per precision (1.09 GB scales, 8.76 GB binary). The paper states this is a 49% reduction in total memory footprint.

• 2-bit accuracy advantage: On LLaMA-3.1-8B, AnyBCQ (multi-precision) reaches 35.32 MMLU and 58.71 CSR average at 2 bits, versus 24.83 MMLU / 45.78 CSR average for ShiftAddLLM, 24.66 / 39.65 for Any-Precision LLM, and 24.12 / 35.60 for AWQ. The fixed-precision AnyBCQ variant scores 35.96 MMLU and 58.89 CSR average.

• Competitive at higher precision: At 3 bits, AnyBCQ (multi-precision) reaches 58.28 MMLU and 69.30 CSR average, compared to Any-Precision LLM at 55.53 / 66.45 and ShiftAddLLM at 56.53 / 69.33. At 4 bits it reaches 63.15 MMLU and 72.37 CSR average, versus 64.04 / 72.57 for Any-Precision LLM. The FP16 baseline is 65.02 MMLU and 72.72 CSR average.

• Multi-precision vs fixed-precision gap: A fixed-precision AnyBCQ variant slightly outperforms the multi-precision one at 3 and 4 bits (for example, 59.41 vs 58.28 MMLU at 3 bits). The paper attributes this to the shared-binary constraint narrowing the optimization space as bit-width increases, and reports perplexity confirming the same pattern (Wiki perplexity 7.68 fixed vs 8.08 multi at 3 bits; 6.62 vs 6.84 at 4 bits).

• Kernel latency: On NVIDIA A100 GPUs with 80 GB HBM and CUDA 12.6, AnyBCQ GEMV kernels beat both cuBLAS (FP16 weights) and the Any-Precision LLM kernel across most shapes. For a layer shape of N=28672, K=8192, cuBLAS takes 2968 µs (×1.00), Any-Precision LLM takes 971 / 1265 / 1348 µs at 2 / 3 / 4 bits (×3.06 / ×2.35 / ×2.20), and AnyBCQ takes 742 / 939 / 1142 µs (×4.00 / ×3.16 / ×2.60).

• Speedup summary: The abstract reports throughput gains of up to 3.0× over half precision and 1.2× over state-of-the-art multi-precision methods.

• End-to-end throughput: Across Llama-3.1-8B, Gemma-2-9b, and Phi-4-14b, AnyBCQ delivers roughly 7-17% higher tokens per second on average than Any-Precision LLM. For example, Phi-4-14b at 2 bits gives 171 vs 147 tokens/sec, and Llama-3.1-8B at 2 bits gives 245 vs 228 tokens/sec (FP16 is 105 and 56 tokens/sec respectively).

• Mixed-precision case study: On CNN/DailyMail abstractive summarization under the Progressive Mixed-Precision Decoding setting, AnyBCQ scores 0.178 ROUGE-L and 0.849 BERTScore at an average precision of 3.6, versus 0.154 and 0.840 for Any-Precision LLM. At an average precision of 2.23, the gap widens to 0.113 / 0.830 versus 0.097 / 0.821. The paper reports that Any-Precision LLM at about 3 average bits degenerates into repeating the same words until the maximum sequence length.

• Where the baseline's overhead lives: A cycle-level breakdown of the Any-Precision LLM kernel using clock64() attributes roughly 35-58% of latency to bit transposition and about 9-17% to LUT centroid lookups, with the remainder in GEMM and memory operations.

Methodology in Plain English

The team starts from Binary-Coded Quantization, which represents a weight matrix as a sum of binary matrices (entries -1 or +1) each multiplied by a real-valued scale factor. This format is convenient because multiplication by a binary plane reduces to adding or subtracting activations, with no centroid table needed.

To make this multi-precision, they build the model hierarchically. They first quantize the weights at a base precision (2 bits in the paper's illustration). At each step up, they freeze the binary codes already assigned, add one new bit-plane derived from the current residual, initialize its scale to zero, and then refine. At the base precision, binary codes and scales are found by a greedy initialization followed by T alternating cycles of least-squares scale updates and binary search code reassignment. At higher precisions, only the newly added bit-plane's codes are assigned (as the sign of the residual) and the scales are updated by least squares; the existing codes are left alone. After initialization, all scaling factors are jointly optimized by minimizing block-wise reconstruction error within each decoder layer.

For inference, the binary bit-planes are shared across precisions, so only one copy is stored. Running at p bits means loading only the first p bit-planes and combining them with the scale set corresponding to p bits. The CUDA kernel fetches a single bit-plane, performs the -1/+1 combination against the activations as adds and subtracts, uses a lookup-table-based GEMM scheme to cache repeated partial results, and accumulates scaled partial sums across planes before returning the output. Because it never loads bits it does not need, lower precision translates directly into proportionally lower memory bandwidth use, which matters in the memory-bound decoding regime.

Experimental setup: LLaMA-3.1-8B evaluated with 5-shot MMLU and zero-shot common-sense reasoning (ARC-Challenge, ARC-Easy, HellaSwag, PIQA, WinoGrande) via lm-eval-harness v0.4.5. Calibration used 512 sequences from C4, 10 epochs, asymmetric BCQ, group size g=128, learning rate 1e-4, and T=20 refinement cycles.

Why This Matters

The paper argues that existing multi-precision approaches are trapped in a 3-4 bit operating range, because 2-bit accuracy collapses in non-uniform schemes, while 4 bits already approaches full precision. If a multi-precision model cannot serve 2 bits usefully, the flexibility it sells is largely nominal. AnyBCQ's main claim to importance is that it makes the low-bit end of the range usable while also removing lookup and transposition overhead, so both accuracy and latency improve at the same time rather than trading off.

Real-world applications:

• Serving LLM inference under varying SLOs, where the same deployed model answers some requests at low precision for speed and others at higher precision for quality.
• Memory-constrained deployment on smaller or shared GPU fleets, where storing one multi-precision model instead of several per-precision checkpoints cuts the storage and memory footprint.
• Mixed-precision decoding pipelines of the kind studied in Progressive Mixed-Precision Decoding and DP-LLM, where precision changes per token or per layer during generation.
• Summarization and other long-generation tasks, where the case study shows the low-bit tail of a decoding schedule can otherwise produce degenerating output.

Industry relevance: the authors are from NAVER Cloud, the code is released at github.com/naver-aics/anybcq, and the paper notes that accelerators such as iFPU and FIGLUT already adopt BCQ-style formats, so a BCQ-based multi-precision model is positioned to run on hardware that supports that family of formats rather than requiring new hardware.

Future Directions

• Theoretical grounding: The paper states that the work "remains largely empirical and lacks theoretical guarantees," and specifically calls for more rigorous analysis of the progressive precision expansion procedure to guide better initialization strategies and bit-allocation schedules.
• Closing the higher-bit accuracy gap: The shared-binary constraint causes the multi-precision model to trail the fixed-precision variant at 3 and 4 bits. Relaxing or better managing that constraint is an open problem the paper identifies.
• Deployment on BCQ-native accelerators: The authors expect larger gains than those observed on conventional GPUs when AnyBCQ is deployed on accelerators like iFPU and FIGLUT that natively support BCQ-style formats, but this was not measured here.
• Wider use in mixed-precision schedulers: The case study only covers the PMPD setting on CNN/DailyMail; extending evaluation to other dynamic precision schedulers such as DP-LLM, and to more models and tasks, is a natural next step.

Target Audience

This paper is most useful for systems and inference engineers deploying LLMs under memory or latency constraints, quantization researchers interested in multi-precision and BCQ-based formats, and kernel or accelerator designers who need a quantized weight format that supports bit-plane-level computation and runtime precision selection. Practitioners already working with 4-bit weight-only quantization will find the 2-bit results and the kernel latency comparisons most directly applicable; readers without any quantization background may need to consult the cited BCQ and GPTQ/AWQ references first.

Authors’ abstract

The deployment of large language models (LLMs) is increasingly constrained by memory and latency bottlenecks, motivating the need for quantization techniques that flexibly balance accuracy and efficiency. Recent work has introduced multi-precision models, which enable inference at multiple precisions within a single model depending on runtime constraints. To support such flexibility, quantized weights are often stored as bit-planes, where hardware efficiency improves when the compute operates directly at the bit-plane level and activates only the precision required by each request. In this work, we present AnyBCQ, a hardware-friendly multi-precision extension of Binary-Coded Quantization (BCQ) that supports direct bit-plane operations. By representing weights as binary bit-planes with corresponding scale factors, AnyBCQ enables bit-plane-level computation and maps naturally to accelerator-friendly, bit-parallel arithmetic. Our progressive precision expansion mechanism incrementally refines scaling factors while reusing previously assigned binary codes, yielding monotonic improvements in accuracy as additional bits are enabled. We further co-design a specialized kernel that exploits the BCQ structure to support dynamic per-request precision selection with negligible overhead. Experiments on recent LLMs demonstrate that AnyBCQ significantly narrows the accuracy drop in the low-bit regime (e.g. 2-bit), remains competitive at higher precision, and achieves throughput gains of up to 3.0x over half precision and 1.2x over state-of-the-art multi-precision methods. By aligning algorithmic flexibility with hardware efficiency, AnyBCQ provides a practical foundation for multi-precision LLM deployment across diverse service-level objectives.