SoloQ: Calibration-Free Quantization for Diffusion Language Models

AI summary

Overview

Research area: Efficient machine learning — specifically post-training quantization (PTQ) for diffusion large language models (dLLMs).

Technical level: Advanced. The paper assumes familiarity with transformer internals, low-bit quantization formats (INT4, NVFP4, E2M1/E4M3), rotation-based quantization (QuaRot, QuIP, RPBH), lattice and Lloyd–Max codebooks, and diffusion language modeling with masked denoising and KV-caches.

Scope: The paper proposes SoloQ, a calibration-free weight–activation–KV quantization framework built on a Kronecker-extended randomized permuted block-Hadamard (K-RPBH) rotation, and evaluates it on both full-sequence dLLMs (LLaDA-Base-8B, LLaDA-1.5-8B, Dream-7B) and block-diffusion dLLMs (Fast-dLLM v2-7B, Nemotron-Labs-Diffusion-8B).

What This Paper Is About

Diffusion LLMs generate text by iteratively unmasking tokens over many denoising steps, and their activation distributions shift substantially across masking states and denoising steps. Existing PTQ methods for dLLMs depend on calibration data, and the paper shows that simply swapping the calibration set (from WinoGrande to MMLU) changes downstream accuracy, meaning the calibration choice does not generalize across tasks. SoloQ's goal is to avoid calibration entirely by transforming weights and activations into a rotated space with a predictable, dimension-dependent marginal distribution that can be quantized data-independently.

Key Contributions

1. A calibration-free dLLM quantization framework (SoloQ) that combines a K-RPBH rotation with a lightweight scale-only bias correction, replacing calibration-set dependence with a single predictable post-rotation distribution.
2. Two quantization paths from one distribution: a distribution-matched codebook variant (SoloQ-C, using a Lloyd–Max scalar quantizer for activations and an E8P lattice quantizer for weights) and a hardware-native NVFP4 variant (SoloQ-N) using the fixed E2M1 format with tensor- and block-wise scaling.
3. Commit-time KV-cache quantization that extends the same calibration-free design to block-diffusion models, compressing only persistent KV states while keeping the actively denoised block in full precision.
4. Empirical validation across five dLLMs showing 4-bit accuracy competitive with or better than calibration-based baselines, with up to 2.61× peak memory reduction and 2.24× end-to-end inference speedup, plus an FPGA implementation described in Appendix E.

Main Findings

• K-RPBH improves distribution matching over RPBH: On a Nemotron representation, the component ablation reports KL divergence dropping from 0.11396 (Block-RHT) to 0.04683 (adding randomized permutation), 0.00682 (adding cross-block mixing) and 0.00413 (K-RPBH with both), with block-energy imbalance falling from 67.89% to 45.64%, 36.24% and 22.12% respectively. Proposition 1 proves that Haar-random cross-block mixing makes the expected conditional variance exactly 1/d.

• Rotation cost vs. accuracy trade-off: Under SoloQ-N on LLaDA-1.5, Haar rotation gives 73.79/63.51 (HellaSwag/MMLU) at 0.559 ms latency, whereas K-RPBH gives 73.86/64.70 at 0.048 ms. On Fast-dLLM v2, Haar gives 57.90/33.48 at 98.135 ms versus K-RPBH at 56.10/33.71 at 5.779 ms. QuaRot is reported as N/A (unsupported dimensions) for Fast-dLLM v2 but reaches 73.41/63.19 at 0.025 ms on LLaDA-1.5.

• Full-sequence W4A4 results (Table 1): SoloQ-C averages 57.74 on LLaDA-Base (a 3.45-point improvement over DLLMQuant++ at 54.29), and SoloQ-N reaches 68.48 on LLaDA-1.5 (a 4.17-point gain over DLLMQuant++ at 64.31). On Dream-7B, SoloQ-N reaches 62.53 versus 61.90 for DLLMQuant++, while STaR-Quant is higher at 63.59. FP16 references are 58.95, 69.23 and 66.54 respectively.

• Block-diffusion results (Table 2): On Fast-dLLM v2 under W4A4, SoloQ-C averages 57.00 and SoloQ-N 56.77, against FP16 60.35, RTN 6.67 and AWQ 6.61, with QuaRot+GPTQ listed as N/A. Under W4A4KV4, SoloQ-C averages 56.19 and SoloQ-N 56.21.

• Block-diffusion results on Nemotron-Labs-Diffusion-8B: Under W4A4, SoloQ-C averages 69.74 and SoloQ-N 70.76, versus FP16 78.67, RTN 1.30, AWQ 1.28 and QuaRot+GPTQ 59.39. Under W4A4KV4, SoloQ-N reaches 71.06 and SoloQ-C 70.18.

• KV-cache quantization (Table 5, 2-bit KV at roughly 3 bits-per-entry with weights/activations in FP16 on Fast-dLLM v2): SoloQ-C averages 53.31 and SoloQ-N 53.53, compared with KIVI 50.26, OScar 46.62, TurboQuant 4.03 and FP16 72.67. SoloQ-C is best on GSM8K and SoloQ-N on IFEval in this comparison.

• Scale-only bias correction helps: On LLaDA-1.5, HellaSwag improves from 72.49/73.71 to 73.01/73.86, and on Fast-dLLM v2 GPQA improves from 30.03/31.92 to 30.58/33.71 when the correction is applied.

• Memory and latency: SoloQ-N achieves a 2.31×–2.61× reduction in peak memory and 1.94×–2.24× end-to-end speedup over BF16 across five dLLMs. For Nemotron, comparing the same BF16-recorded sequence of 75 forward passes yields a 2.05× speedup. SoloQ-C uses 8-bit Tensor Core kernels on H200 and SoloQ-N uses native NVFP4 kernels on an RTX Pro 6000.

• Reported reproducibility caveat: The paper notes that TruthfulQA results for DLLMQuant/STaR-Quant are not reproducible because they are not open-sourced, and that their FP is 49.76 versus 43.96 in the authors' setup.

Methodology in Plain English

The starting observation is that dLLM activations change shape as the model moves from heavily masked states to nearly complete sequences, so any quantizer tuned on calibration samples risks being wrong for other states. SoloQ sidesteps this by rotating every weight and activation vector into a common basis where each coordinate behaves like a sample from a known, nearly Gaussian distribution that depends only on the vector's dimension. Because that distribution is known analytically, quantizer levels can be designed in advance rather than fitted to data.

Two ingredients make the rotation practical. First, the rotation itself must not be prohibitively expensive, so the paper builds it from a random permutation, small Hadamard blocks (which can be computed quickly, in O(d log h) operations), and a small Haar-random orthogonal matrix that mixes information across blocks. That cross-block mixing step is what fixes the residual energy imbalance left when a vector is split into many blocks — a real issue for wide dimensions such as Dream's FFN dimension of 18944 = 512 × 37. Second, low-bit rounding tends to shrink vectors slightly, so a single per-token or per-row scale factor is computed to restore the component of each vector along its original direction, without needing extra bits or side information.

With that normalized rotated representation, the paper quantizes in one of two ways: a codebook path where activations use a Lloyd–Max quantizer and weights use an E8P lattice quantizer, or a hardware-native path using NVFP4 with E2M1 values grouped over 16 coordinates and E4M3 block scales. For block-diffusion models, which keep KV states from already-committed blocks, keys are quantized per-channel asymmetrically in KIVI style and values follow the SoloQ activation path — but only at the moment a block is committed, so the block currently being denoised is never quantized.

Why This Matters

Impact on research: The paper shifts the dLLM quantization question from "which calibration set best represents the shifting distribution?" to "how do we reshape the distribution so no calibration set is needed?" That reframing matters because the paper directly demonstrates that calibration-set choice (WinoGrande vs. MMLU) changes benchmark outcomes, and because it extends calibration-free quantization to persistent KV states in block-diffusion models — a setting the authors state is not previously covered.

Real-world applications (as supported by the paper's evaluations):

• On-device or edge deployment of diffusion LLMs, where calibration datasets may be unavailable or privacy-sensitive, and where the reported 2.31×–2.61× peak memory reduction enables 4-bit model storage.
• Long-context generation with block-diffusion models, where the paper profiles KV-cache and activation memory growing with context length and targets that growth with commit-time KV quantization.
• Latency-sensitive serving, where the reported 1.94×–2.24× end-to-end speedup over BF16 (and 2.05× for the fixed 75-forward-pass Nemotron comparison) reduces the cost of repeated denoising forward passes.
• Hardware-specific deployment, since SoloQ-N maps onto native NVFP4 kernels on an RTX Pro 6000 and the paper also reports an FPGA implementation (Appendix E), while SoloQ-C uses 8-bit Tensor Core kernels on H200.

Industry relevance: Because the method requires no calibration pass and needs no pre-collected activation statistics, it fits cleanly into production quantization pipelines where calibration data collection is a bottleneck. The dual codebook/NVFP4 support also means the same framework can target both research-oriented and hardware-native deployment paths.

Future Directions

• Hardware and kernel co-design: The paper reports SoloQ-C on H200 8-bit Tensor Core kernels, SoloQ-N on RTX Pro 6000 NVFP4 kernels, and an FPGA implementation in Appendix E; broadening this to other accelerators and to the codebook path's runtime cost is a natural extension.
• Filling the block-diffusion baseline gap: The authors note that prior quantization results are limited for block-diffusion models, that QuaRot cannot support Fast-dLLM v2's dimensions, and that most block-diffusion benchmarks are generation-based rather than likelihood-based. Establishing a wider, more directly comparable baseline suite remains open.
• Extreme low-bit regimes: The paper evaluates W4A4 and W4A4KV4 plus a 2-bit KV comparison; whether the predictable post-rotation distribution holds up at more aggressive weight/activation bit-widths is left open.
• Reconciling accuracy gaps: On Dream-7B, STaR-Quant averages 63.59 versus SoloQ-N's 62.53, and the paper's own footnote flags a non-reproducible TruthfulQA discrepancy (49.76 reported by others vs. 43.96 in the authors' setup). Understanding these gaps, and the limitations listed in Appendix G, are open threads.

Target Audience

This paper is most useful to quantization and efficient-inference researchers working on post-training compression of large models, to systems and kernel engineers deploying diffusion LLMs under memory or latency constraints, and to hardware architects evaluating NVFP4-class formats and FPGA mappings. Readers without a background in rotation-based quantization or diffusion language modeling will find the preliminary and method sections necessary before the results tables are interpretable.

Authors’ abstract

Diffusion large language models dLLMs) have emerged as a promising alternative to autoregressive language models through bidirectional diffusion-based token generation. However, their growing model sizes and high inference costs make efficient deployment challenging: full-sequence denoising repeatedly invokes compute-intensive forward passes, while block-diffusion models additionally introduce a memory-intensive KV-cache. Low-bit weight-activation quantization is therefore attractive, yet existing dLLM post-training quantization methods rely on calibration data despite activation distributions shifting across masking states and denoising steps. We present SoloQ, a calibration-free quantization framework that maps weights and activations into a normalized rotated basis with a predictable marginal distribution, enabling data-independent quantization. SoloQ combines a structured K-RPBH rotation with a lightweight rescaling correction for calibration-free quantization. Its predictable post-rotation distribution supports both distribution-matched codebooks and hardware-native NVFP4. For block-diffusion models, SoloQ further applies commit-time KV-cache quantization to compress persistent states without perturbing the actively denoised block. Across full-sequence dLLMs (LLaDA and Dream) and block-diffusion dLLMs(Fast-dLLM v2 and Nemotron-Labs-Diffusion), SoloQ retains accuracy under 4-bit quantization and outperforms calibration-based baselines on knowledge- and reasoning-intensive benchmarks. With NVFP4, SoloQ reduces peak memory by up to 2.61X and accelerates end-to-end inference by up to 2.24X.