RaBiT: Residual-Aware Binarization Training for Accurate and Efficient LLMs

AI summary

Overview

Research area: Extreme low-bit quantization of large language models, specifically 2-bit residual binarization and quantization-aware training (QAT).

Technical level: Advanced. The paper combines a formal MSE decomposition with correlation analysis, a custom coupled training loop, and hardware kernel benchmarks; readers will need familiarity with quantization, straight-through estimators, and knowledge distillation.

Scope: The paper diagnoses a training pathology in stacked binary (residual) LLM quantization and proposes a single-shared-weight training framework that enforces an error-correction hierarchy between binary paths.

What This Paper Is About

Residual binarization stacks multiple binary (±1) weight layers to reach 2-bit precision while keeping inference free of matrix multiplications, but the stack's promised error-compensation behavior breaks down during training. The authors show that a single shared gradient applied to independent latent weights causes parallel binary paths to learn redundant features instead of correcting each other's errors, a failure mode they call inter-path adaptation. RaBiT replaces the independent latent weights with one shared full-precision weight from which each binary path is sequentially derived on the fly, so every path is structurally forced to correct the residual error of the path before it.

Key Contributions

1. Diagnosis of inter-path adaptation. The authors identify and analyze inter-path adaptation as a specific manifestation of feature co-adaptation in residual binarization, in which the intended error-compensation structure collapses during Standard QAT as parallel paths become functionally redundant.

2. The RaBiT training framework. They propose a QAT framework that resolves inter-path adaptation by enforcing residual coupling on the fly from a single shared full-precision weight, rather than constraining the solution space with heuristics such as path freezing. The design halves the training memory footprint and is coupled with a robust function-aware initialization.

3. Cachet of the 2-bit frontier. The paper reports state-of-the-art 2-bit accuracy, a 4.49× inference speed-up, and competitive performance against hardware-intensive Vector Quantization (VQ) methods despite using matmul-free operations.

4. Code release. Code is available at github.com/SamsungLabs/RaBiT.

Main Findings

• Standard QAT fails to build a residual hierarchy. In the MSE decomposition on Llama2-7B, Standard QAT produces a path correlation close to zero (-0.0752 in Layer 5, -0.1240 in Layer 15, -0.1279 in Layer 25) and residual alignment of 0.4395, 0.4082, and 0.4668 respectively, indicating the parallel paths do not meaningfully reduce the total error.

• Heuristic splitting forces anti-correlation but not true error correction. DB-LLM attains strongly negative path correlations (-0.4941, -0.4570, -0.4824 across the three layers) yet poor residual alignment (0.2617, 0.2500, 0.2490), showing that mechanical anti-correlation can be induced without the second path tracking the actual functional residual. MBOK improves residual alignment over Standard QAT (0.4805, 0.4570, 0.5039) but yields only weak anti-correlation and limited loss-reducing covariance.

• RaBiT achieves the highest residual alignment. RaBiT reaches 0.6484 (Layer 5), 0.5820 (Layer 15), and 0.6172 (Layer 25), with the lowest total MSE in every reported layer (0.0009, 0.0094, 0.0327), confirming that deriving the second path from the dynamic residual produces genuine error cancellation.

• State-of-the-art 2-bit Llama results. On Llama2-7B, RaBiT reaches 5.78 WikiText-2 perplexity and 7.64 C4 perplexity with 61.51% average zero-shot QA, versus MBOK (6.99 PPL, 53.63% QA) and DBF (6.10 PPL, 58.42% QA); it also edges out the VQ method QTIP (5.86 PPL, 58.97% QA). On Llama2-13B, RaBiT records 5.15/6.95 PPL and 62.10% QA; on Llama3-8B, 7.34/10.52 PPL and 64.13% QA.

• Robustness where baseline binarization collapses. BitStack degrades catastrophically on Llama3-8B (2.75e3 WikiText-2 PPL, 36.21% QA) and QuIP# degrades to 8.70 PPL, while RaBiT maintains 7.34 PPL and high fidelity.

• Strong results beyond the Llama family. On Gemma3 (1B/4B/12B), RaBiT records 11.27/15.54 (53.18% QA), 8.09/11.91 (62.21% QA), and 6.66/10.18 (68.85% QA). On Qwen3 (1.7B/4B/8B) it records 10.19/14.49 (60.38% QA), 8.27/12.08 (66.66% QA), and 7.47/11.00 (70.25% QA). The paper describes these as state-of-the-art on Gemma3 1B/4B/12B, Qwen3-1.7B and Qwen3-4B, and highly competitive on Qwen3-8B.

• Better preservation on harder reasoning tasks. Across BBH, GPQA, MMLU-Pro and IFEval, RaBiT averages 27.14 versus QTIP's 25.38 on Llama2-13B (baseline 29.27), and 25.12 versus QTIP's 23.92 on Llama3-8B (baseline 31.03).

• Efficiency at both kernel and system level. Using specialized packed kernels on an NVIDIA RTX 4090, 2-bit RaBiT delivers 7.72 μs (2.22

Authors’ abstract

Efficient deployment of large language models (LLMs) requires extreme quantization, forcing a critical trade-off between low-bit efficiency and performance. Residual binarization enables hardware-friendly, matmul-free inference by stacking binary ($\pm$1) layers, but is plagued by pathological feature co-adaptation. We identify a key failure mode, which we term inter-path adaptation: during quantization-aware training (QAT), parallel residual binary paths learn redundant features, degrading the error-compensation structure and limiting the expressive capacity of the model. While prior work relies on heuristic workarounds (e.g., path freezing) that constrain the solution space, we propose RaBiT, a novel quantization framework that resolves co-adaptation by algorithmically enforcing a residual hierarchy. Its core mechanism sequentially derives each binary path from a single shared full-precision weight, which ensures that every path corrects the error of the preceding one. This process is stabilized by a robust initialization that prioritizes functional preservation over mere weight approximation. RaBiT redefines the 2-bit accuracy-efficiency frontier: it achieves state-of-the-art performance, rivals even hardware-intensive Vector Quantization (VQ) methods, and delivers a $4.49\times$ inference speed-up over full-precision models on an RTX 4090. Code is available at https://github.com/SamsungLabs/RaBiT.