RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention

AI summary

Overview

Research area: Efficient inference and memory compression for large mixture-of-experts (MoE) language models, with a focus on on-device (Apple Silicon) deployment.

Technical level: Advanced. The paper combines quantization theory (Walsh–Hadamard transforms, SO(4) quaternion parameterization, Hanson–Wright concentration bounds) with low-level GPU kernel engineering in Metal and llama.cpp.

One-sentence scope: The paper introduces RotaryQuant, a three-axis compression system whose novel component, IsoQuant, performs attention directly on 3-bit packed key-value data to fit MoE models from 26B to 120B parameters inside 16–32 GB memory budgets without meaningful quality loss.

What This Paper Is About

Running large MoE language models locally is blocked by three memory pressures at once: resident weight matrices, a key-value (KV) cache that grows linearly with context, and dozens of expert sublayers that must be paged on demand. The paper's goal is to fit models like Gemma 4-26B-A4B, Qwen3-30B-A3B, and Nemotron-H 120B onto consumer devices with 16–32 GB of unified memory while keeping interactive speed and near-lossless output quality.

Key Contributions

1. IsoQuant rotation. A structured rotation combining a Walsh–Hadamard transform with block-diagonal SO(4) rotations that isotropizes activation distributions before 3-bit scalar quantization. It requires O(d log d) operations and 256 stored parameters per head, versus O(d²) and 16,384 parameters for dense rotation methods — which the paper describes as 64× fewer stored parameters.

2. Fused compressed-space decode. A four-kernel Metal GPU pipeline that performs QK dot products and value accumulation directly on packed 3-bit KV data, eliminating full-precision tensor materialization during autoregressive decode. The inverse rotation is applied once to the aggregated output rather than per token.

3. RotaryQuant: a three-axis memory system. A principled composition of mixed-precision weight quantization, IsoQuant KV compression, and LRU expert offloading, enabling a 120B-parameter MoE to run with 17.2 GB peak memory on a 32 GB Apple M4 Max.

4. Comprehensive evaluation. Near-zero quality degradation (ΔPPL < 0.002) across three architecturally distinct models, plus decode profiling that identifies when KV compression helps and when it does not.

Main Findings

• KV fidelity versus dense rotation: At 2,048-token context on Qwen3-30B-A3B (default PPL 1.0844), IsoQuant adds only +0.0009 ΔPPL while TurboQuant adds +0.0405 — a 45× lower degradation. On Nemotron-H 120B (default PPL 1.0866), IsoQuant adds +0.0012 versus +0.0039 for TurboQuant. On Gemma 4-26B-A4B (default PPL 1.3483), IsoQuant adds +0.0000 versus +0.0622, a result the paper attributes to only 5 of 30 layers using global attention and therefore engaging the compressor.

• End-to-end memory and speed: Gemma 4-26B-A4B runs at 12.85 tok/s with 5.4 GB peak (16 GB budget); Nemotron-H 120B at 14.85 tok/s with 17.2 GB peak (32 GB budget); Qwen3.6-35B-A3B at 15.6 tok/s with 6.8 GB peak (16 GB budget). All three pass the 12/12 quality gate; Gemma 4 and Nemotron-H pass a 2-hour soak test, while the Qwen3.6 soak test was not completed because the model was released after the initial soak campaign. The abstract frames throughput as 9–19 tok/s.

• Compression versus speed trade-off: On Gemma 4 without expert offloading, IsoQuant costs 5.3× throughput (20.6 tok/s versus 109.8 tok/s for FP16 KV) at similar peak memory (10,748 MB versus 10,649 MB). Expert offloading alone imposes a roughly 100× penalty (1.01 tok/s) regardless of KV mode. The paper concludes the system is a memory-pressure relief mechanism, not a universal accelerator.

• When KV compression pays off: KV attention is 51% of decode time on Gemma 4, 54% on Qwen3, but only 14% on Nemotron-H 120B, whose Mamba layers bypass the KV cache. The paper's deployment criterion: IsoQuant is most beneficial when KV attention exceeds about 20% of decode time.

• Rotation design ablation: Single-quaternion conjugation (SO(3)) passed 0/5 quality gates and left 25% of dimensions unmixed; block SO(4) without the WHT passed 1/5 and failed to capture global correlations; the WHT + SO(4) composition passed 5/5.

• Stability: Over three independent runs with full model reload on Gemma 4-26B-A4B, IsoQuant's coefficient of variation was 0.3% (1.05 mean tok/s, 0.003 std dev) versus 5.4% for the default FP16 path (1.01 mean tok/s, 0.055 std dev) — 18× lower. IsoQuant added about 359 MB peak memory with no throughput advantage in the offload regime.

• Zero materialization verified: Runtime instrumentation over 3,612 decode steps recorded fused_metal_success_rate 1.000, decompress_calls 0, read_keys_calls 0, fallback_invocations 0, and packed_cache_hit_rate 0.000.

• llama.cpp portability: As a new KV type (GGML_TYPE_ISOQUANT3_0), the fused path reached 4,093.8 prompt t/s (−0.5%) and 96.98 generation t/s (−3.2%) versus a TurboQuant 3-bit baseline of 4,114.6 and 100.15. The unfused path degraded to 2,306.2 (−44%) and 81.92 (−18%), eliminating 280 extra dispatch launches when fused.

• Long context: On Qwen3.6-35B-A3B with nvfp4 weights and IsoQuant 3-bit KV, WikiText-103 perplexity at 32,768 tokens was 5.624 versus a 5.625 baseline (0.024% divergence, against a 5% pass threshold). A Needle-in-a-Haystack test with 4 random numeric needles at 50% depth achieved 100% accuracy at 4K, 8K, 16K, and 32K contexts.

• Inverse rotation is mandatory: Skipping it caused perplexity to explode from 7.05 to 15,369.

• Negative results reported: An attention-residual (AttnRes) predictor for cross-layer expert prefetching caused −10.6% to −11.2% throughput regression with 0% hit-rate improvement over the LRU baseline, attributed to CPU/GPU command buffer contention on Apple Silicon's unified memory. Task-aware expert pinning also showed no benefit.

• Serving limitations: Under concurrent load, the MLX-based server degraded — at 8 concurrent clients, 4/8 responses returned empty bodies despite HTTP 200 status codes, and throughput plateaued by 4 clients at 34% parallel efficiency.

• MLA/RoPE blocker: Models with Multi-Head Latent Attention split KV into content and positional sub-spaces; IsoQuant does not yet implement the required sub-block split.

Methodology in Plain English

The authors attack three memory consumers in a fixed order, since each choice constrains the next: weight quantization determines the activation distributions the KV compressor must handle, and expert offloading depends on the leftover memory budget.

For weights, they assign precision by architectural role rather than uniformly: 4-bit for dense layers (GPTQ/AWQ), 2-bit for routed experts, and 8-bit (Q8_0) for the shared expert, because the shared expert has much higher activation kurtosis (κ = 10.10 versus 0.41 for specialist experts) and resists aggressive compression. Non-resident experts are paged to disk with an LRU policy and loaded on demand.

For the KV cache, each 128-dimensional key or value vector goes through four stages: normalize to the unit sphere and store the norm as a scale; apply a rotation composed of a normalized Walsh–Hadamard matrix followed by independent SO(4) rotations on each contiguous 4-element block; quantize each coordinate to 3 bits using Lloyd–Max optimal codebooks; and bit-pack. At 3 bits, a 128-dimensional vector packs into 48 bytes versus 256 bytes in FP16, roughly 5.3× compression. The rotation is cheap because the WHT has a butterfly structure (896 FMAs at d = 128) and each SO(4) block uses 16 FMAs (512 total), giving 1,408 FMAs versus 16,384 for a dense 128×128 rotation — an 11.6× reduction. Storage is (d/4)×6 = 192 learnable rotation parameters plus 64 codebook entries, matching the 256-per-head figure.

Because the rotation is orthogonal, inner products are preserved: the query is rotated forward, attention scores are computed against rotated keys, values are aggregated in the rotated space, and a single inverse rotation recovers the exact pre-quantization output. That property is what lets the Metal kernels skip decompression entirely — a four-kernel pipeline (fused QK dot on packed K, standard softmax, fused value accumulation on packed V, and a structured inverse rotation) computes attention on 3-bit data. Profiling showed value accumulation dominates decode time at about 0.79 ms, addressed by a dual-strategy kernel: word-parallel dispatch below 512 tokens, dimension-parallel at or above 512.

During prefill, KV vectors are stored in FP16 and compressed in bulk at the prefill-to-decode boundary, avoiding compounding quantization error across prompt positions; the transient buffer costs about 230 MB at L = 28 layers, H_kv = 8 heads, d = 128, and T = 2,048 tokens.

Evaluation used four architecturally distinct models — Gemma 4-26B-A4B, Qwen3-30B-A3B, Qwen3.6-35B-A3B, and Nemotron-H 120B — on a single Apple M4 Max with 128 GB unified memory, with a 12-prompt automated quality gate (greedy decoding, temperature 0, seed 42, max tokens 500, repetition ratio ≤ 0.22 in strict mode).

Why This Matters

Impact on research: The paper reframes KV cache compression as an execution-model question rather than only a storage question. Existing methods (KIVI, KVQuant, QJL, TurboQuant, SpinQuant, QuaRot) reconstruct full-precision tensors before standard GEMM-based attention; this work argues that materialization negates much of the memory saving and proposes a compressed-domain analogue of FlashAttention's tiling idea. Its structured-rotation alternative also shows that a WHT plus block rotations can beat a dense learned rotation in quality on the tested models while costing far less compute and storage.

Real-world applications:

• On-device assistants and private chat: prompts never leave the user's machine, eliminating the privacy exposure of cloud inference.
• Low-connectivity and low-income users: no recurring API subscription and no bandwidth requirement, addressing the accessibility angle the paper raises.
• Environmentally motivated deployments: the authors note consumer SoCs have lower per-inference power draw than datacenter GPUs.
• Local developer tooling and coding assistants built on MoE checkpoints, validated through the llama.cpp integration.

Industry relevance: The work targets a practical gap — models otherwise requiring datacenter hardware now run on a laptop within stated budgets. Its honest negative results (expert offloading's ~100× penalty, the AttnRes predictor regression, server concurrency failures at 8 clients) give deployers a realistic picture of when the system helps and when it does not. The paper also flags that unmonitored local deployment bypasses API-level safety filtering and usage monitoring, while noting IsoQuant does not alter model capabilities.

Future Directions

• Extending the sub-block split required for MLA/RoPE architectures, which currently blocks models such as DeepSeek-V2.
• Comprehensive perplexity sweeps across all models beyond 2K tokens, since the main Table 3 measurements are validated at 2K and long-context work is described as preliminary.
• Optimizing the Kernel C value-accumulation bottleneck, which dominates decode time at about 0.79 ms.
• Exploring Mojo kernel prototypes for cross-platform portability beyond Apple Silicon, and building a purpose-built inference server to replace the MLX-based one that degraded under concurrency.

Target Audience

Researchers and engineers working on LLM inference efficiency, KV cache compression, and quantization; systems developers targeting Apple Silicon, Metal, MLX, or llama.cpp; and practitioners trying to deploy 26B–120B MoE models on consumer or edge hardware under fixed memory budgets. Readers need comfort with linear algebra (orthogonal transforms, quaternion parameterizations) and GPU kernel concepts to follow the method sections fully.

Authors’ abstract

Large mixture-of-experts (MoE) language models with 26--120 billion parameters exceed the memory capacity of consumer devices through three simultaneous pressures: resident weight matrices, key-value (KV) cache state that grows linearly with context, and dozens of expert sublayers that must be paged on demand. We present RotaryQuant, a three-axis compression system that addresses all three. Mixed-precision weight quantization assigns bit-widths by architectural role: 4-bit for dense layers, 2-bit for routed experts, and 8-bit for the shared expert whose high activation kurtosis resists aggressive compression. LRU expert offloading pages non-resident experts to disk under genuine memory pressure. The novel axis is IsoQuant, a KV cache compression method that applies a Walsh--Hadamard transform followed by block-diagonal SO(4) rotations to isotropize activation distributions before 3-bit scalar quantization, requiring $O(d \log d)$ operations and 256 stored parameters per head versus $O(d^2)$ and 16{,}384 for dense rotation methods. A fused four-kernel Metal GPU pipeline performs attention directly on packed 3-bit tensors without materializing full-precision KV state---a different execution model, not just a quantization scheme. The combined system fits Gemma 4-26B-A4B and Qwen3-30B-A3B within a 16\,GB budget and Nemotron-H 120B within 32\,GB, running interactively at 9--19 tok/s with near-zero perplexity degradation ($Δ$PPL $\leq +0.0012$) and 100\% retrieval accuracy at 32K context.