DictLLM: Post-training Compression of Large Language Model with Dictionary Kernels

AI summary

Overview

Research area: Model compression for large language models (post-training low-rank factorization, sparsification, and low-bit quantization), with a focus on memory-bound mobile and edge deployment.

Technical level: Advanced. The paper assumes familiarity with singular value decomposition, Hessian-based (second-order) pruning such as SparseGPT, and block floating-point formats such as NVFP4.

Scope: The paper proposes and evaluates DictLLM, a four-step post-training pipeline that represents grouped transformer weight matrices with a shared dictionary plus layer-specific sparse coefficients, then quantizes both factors.

What This Paper Is About

Deploying multi-billion-parameter LLMs on mobile hardware is limited by memory capacity and memory bandwidth rather than by arithmetic. Existing post-training compressors attack redundancy along separate axes — low-rank factorization shrinks matrix dimensions, pruning removes individual weights, and quantization lowers bit width — but they are usually applied in isolation, and prior dictionary-based ("parameter sharing") transformers require training the shared representation from scratch, which is impractical at the billion-parameter scale.

DictLLM's goal is to build a dictionary-based representation directly from a pretrained checkpoint, with no retraining, so that layers within a group share one dictionary kernel while retaining layer-specific sparse coefficients.

Key Contributions

1. Dictionary-based parameter sharing as a post-training method. Weight matrices in a layer group are decomposed into a shared dictionary kernel and layer-wise sparse coefficient matrices, producing a zero-shot post-training compressor that works on billion-parameter LLMs.

2. A three-stage "3+1" construction pipeline. Activation-aware factorization (input-scaled SVD on concatenated grouped weights) followed by dictionary-aware Hessian pruning of the coefficients and then a dictionary refit to the resulting sparse support — improving reconstruction without changing sparsity or model size.

3. Hessian-aware NVFP4 quantization of both factors. The dictionary is quantized using the input covariance, and the coefficients are quantized using the dictionary-induced Hessian, with error compensation and the pruning mask preserved during coefficient quantization.

4. A reported 5.1× reduction of LLaMA-7B with a 2.9-point WikiText-2 perplexity increase, described as a 3.6× higher compression ratio than the state-of-the-art low-rank method.

Main Findings

• Language modeling and reasoning gains on LLaMA-7B. At 1.25× compression, DictLLM reaches WikiText-2 / PTB / C4 perplexity of 6.89 / 12.27 / 11.75 and average zero-shot accuracy of 0.51, versus Basis Sharing at 7.74 / 17.35 / 15.03 and 0.48, and versus the original model at 5.68 / 8.35 / 7.34 and 0.52.

• Robustness at aggressive compression. At 2× compression, DictLLM reports 10.59 / 35.67 / 29.12 perplexity and 0.43 average accuracy, while Basis Sharing at 2× reports 19.99 / 126.35 / 88.44 and 0.36. Basis Sharing is compared at 1.67× with 0.40 average accuracy, against DictLLM's 0.43 at 2×.

• FP4-quantized variants remain competitive. DictLLM FP4 at 5× compression obtains 8.48 WikiText-2 perplexity and 0.47 average accuracy; at 7× it obtains 11.92 and 0.40 — the 0.40 average accuracy at 7× is reported as outperforming Basis Sharing's 0.36 at 2×.

• Pareto frontier claim. DictLLM reports a 5.1× compression of the original LLaMA model with a 2.9-point WikiText-2 perplexity increase, and a 9.4-point WikiText-2 perplexity reduction relative to Basis Sharing at 2× compression.

• Prune the coefficients, not the dictionary. In an ablation at an overall compression gain of 1.43×, pruning only the coefficients (19.4% sparsity) gives the lowest perplexity, 7.83, versus 9.25 for fully dense, 9.04 for pruning the dictionary at 35.0% sparsity, and 7.99 for pruning both at 12.5%. The interpretation is that the dictionary holds information shared across layers, while coefficients carry more layer-specific redundancy.

• Lazy Hessian updates match full inversion at lower cost. On a 20% factorized model pruned to 50% coefficient sparsity, magnitude pruning gives 47.43 perplexity in 44.6 min, full inversion gives 9.89 in 13.49 h, and the dictionary-aware lazy update gives 9.89 in 55.6 min — the same perplexity as full inversion while being 14.6× faster.

• Dictionary refitting helps more as sparsity rises. Refitting the dictionary after pruning reduces WikiText-2 perplexity by 1.04 points at 50% sparsity but only 0.06 points at 20% sparsity, across a 20%–50% sparsity sweep.

• Staged pipeline beats k-SVD. On the first LLaMA-7B query projection with 256 WikiText-2 calibration samples, the factorize–prune–update pipeline achieves lower activation reconstruction error than k-SVD with an undercomplete dictionary while running 3.4× faster.

• All-NVFP4 is the most compact setting tested. With a baseline of 30% factorization reduction and 50% coefficient sparsity (FP16/FP16 at 12.98 perplexity, 1.00×), NVFP4/FP16 gives 13.66 at 1.61×, NVFP4/FP8 gives 14.71 at 2.60×, NVFP4/INT8 gives 16.05 at 2.60×, and NVFP4/NVFP4 gives 16.82 at 3.56×.

• Generalization across model families. At 1.25× compression on WikiText-2: LLaMA2-7B 6.93 (Basis Sharing 7.70, SVD-LLM 8.5, ASVD 10.10); OPT-6.7B 11.36 (11.79, 16.04, 82.0); Mistral-7B 6.46 (7.57, 10.21, 13.72).

• Post-compression LoRA fine-tuning helps both methods. With LoRA applied to query and value weights, DictLLM reports 6.42 / 7.54 / 7.91 / 8.71 versus Basis Sharing's 7.09 / 8.21 / 10.44 / 10.95 at 1.25× / 1.43× / 1.67× / 2×. LoRA is reported to reduce WikiText-2 perplexity by up to 1.9 points and to be more beneficial at higher compression ratios.

• Small calibration budgets suffice. Sweeping calibration size from 32 to 256 samples at 1.43× compression, DictLLM with only 32 samples is reported to outperform Basis Sharing with 256 samples.

Methodology in Plain English

The method treats a group of transformer layers as sharing one reusable "basis" while each layer keeps its own sparse mixing weights, and builds this in four stages from a pretrained model.

1. Factorize. For a group of layers, concatenate their weight matrices and compute a rank-k SVD on the input-scaled version, using input statistics gathered by running 256 WikiText-2 calibration samples (sequence length 2048) through the original model. The shared dictionary comes from the scaled left singular vectors, and the right singular vectors become the per-layer dense coefficients. Sharing one rank budget across several layers means the rank is amortized rather than paid per layer.

2. Prune the coefficients. With the dictionary fixed, sparsify the coefficients to a target sparsity. Because pruning a coefficient entry perturbs the weight by a structured amount through the dictionary, the correct curvature is a dictionary-projected Hessian (dictionary transpose times grouped input covariance times dictionary) rather than the raw input covariance used by SparseGPT on dense weights. Working in this k×k coefficient space rather than the full d_in×d_in input space makes the inverse cheaper (k³ instead of d_in³), and the inverse is shared across all columns, so each column update costs k². A blockwise lazy update avoids recomputing the inverse while matching full-inversion accuracy.

3. Refit the dictionary. Once coefficients are sparse, solve a least-squares problem to update the shared dictionary to the surviving support, keeping the sparsity pattern and coefficient values fixed so the model size does not change.

4. Quantize. Convert both the dictionary and the coefficients to NVFP4 (blocks of FP4 values with a shared microscale), using the input covariance as the Hessian for the dictionary and the dictionary-induced Hessian for the coefficients, with error compensation and masked entries held at zero. The quantization implementation follows QuTLASS.

Experiments use LLaMA-7B, LLaMA2-7B, OPT-6.7B, and Mistral-7B checkpoints from HuggingFace, evaluated on WikiText-2, PTB, and C4 perplexity plus seven zero-shot benchmarks (OpenBookQA, WinoGrande, HellaSwag, PIQA, MathQA, ARC-Easy, ARC-Challenge) via LM-Evaluation-Harness, against ASVD, SVD-LLM, and Basis Sharing. Compression is measured from the number of nonzero entries in the dictionary and coefficient matrices, and runs are on dual NVIDIA RTX PRO 6000 Blackwell GPUs.

Why This Matters

Impact on research. The paper argues that the three compression axes — low-rank structure, sparsity, and low bit width — are complementary but hard to stack naively, because each stage changes the statistics the next stage relies on. DictLLM offers a concrete recipe for composing them in a single representation, and shows that dictionary-based sharing, previously a training-time architecture choice, can be produced post hoc from a checkpoint. It also introduces a dictionary-projected Hessian as the right saliency metric for coefficients, and demonstrates that pruning is better applied to the per-layer coefficients than to the shared dictionary.

Real-world applications:

• On-device assistants and keyboard/autocomplete models running on mobile GPUs with limited memory.
• Edge inference for robotics, wearables, or automotive systems where weights must be fetched repeatedly and memory traffic dominates energy.
• Privacy-preserving local inference, where prompts and data never leave the device.
• Serving scenarios where reducing weight memory traffic improves throughput under a roofline-style bandwidth limit.

Industry relevance. The pipeline requires no pretraining and only a small calibration set (32 to 256 samples), uses a hardware-supported FP4 format, and produces a continuous compression curve controlled by two knobs — dictionary size (via the factorization ratio) and coefficient sparsity (20%–50%). Those properties map directly onto deployment workflows where a single pretrained checkpoint must be repackaged for several device tiers.

Future Directions

• Extending beyond the tested layer grouping and weight types. The paper groups layers for a shared dictionary and reports results for four 6–7B model families; how the grouping granularity and rank allocation should scale to much larger models is not settled.

• Combining more axes within the same Hessian framework. DictLLM quantizes with NVFP4 and prunes coefficients; whether activations, KV-cache, or attention-specific structures can be brought under the same dictionary-induced Hessian formulation remains open.

• Making the refit and quantization steps jointly optimal. The pipeline is deliberately sequential because joint optimization would be a large discrete-continuous problem; the reported gains from refitting (larger at higher sparsity) suggest further alternating updates could be worthwhile if their cost is controlled.

• Validated on-device speedups. The paper reports compression ratios, perplexity, and accuracy, and motivates the work with memory-bandwidth limits, but end-to-end latency and energy measurements on mobile hardware are not reported and would be a natural next step.

Target Audience

Researchers and engineers working on LLM compression and efficient inference — particularly those already familiar with SVD-based low-rank methods, SparseGPT-style second-order pruning, and FP4/NVFP4 quantization — as well as systems practitioners who need to fit 7B-class models into tight memory budgets without retraining. Readers new to compression will find the high-level framing accessible, but the Hessian derivations and quantization details assume a strong background.

Authors’ abstract

Efficient deployment of large language models (LLMs) on mobile devices requires balancing performance with compute and memory constraints. Post-training compression is an effective approach to managing this trade-off, while requiring much less computation than in-training counterparts. We propose DictLLM, a dictionary-based compression scheme in which weight matrices within grouped layers are decomposed into a shared dictionary and layer-specific sparse coefficients, directly obtained from a pre-trained model without further training. Specifically, we cast DictLLM as a four-step post-training optimization. First, we obtain shared dictionaries via singular value decomposition (SVD) of input-scaled weight matrices, amortizing rank across grouped layers; second, we sparsify the layer-specific coefficients to the target compression ratio using a dictionary-aware Hessian; and third, we refine the dictionaries given the resulting sparse coefficients. Finally, we aggressively quantize both components using factor-specific Hessian information: input covariance for the dictionaries and the dictionary-aware Hessian for the coefficients. The resulting representation reduces memory footprint by storing a single shared dictionary per layer group, along with sparse layer-specific coefficients. On language modeling and zero-shot reasoning benchmarks, DictLLM consistently outperforms existing post-training low-rank compression methods and pushes the Pareto frontier between model size and performance. DictLLM compresses the original LLaMA-7B model by 5.1x with only a 2.9-point increase in WikiText-2 perplexity, achieving a 3.6x higher compression ratio than the state-of-the-art low-rank method.