Advancing SVD-based LLM Compression via Layer-Wise Error Model Search

AI summary

Overview

Research area: Efficient machine learning — specifically post-training compression of large language models (LLMs) via low-rank singular value decomposition (SVD), covering both how each weight matrix is decomposed and how many ranks each layer receives under a fixed parameter budget.

Technical level: Intermediate. The paper assumes familiarity with SVD, second-order/Fisher approximations of the loss, and the standard "calibration data + reconstruction error" framing of post-training compression.

Scope (one sentence): The paper introduces two complementary components — KFAC-SVD, a token-wise Fisher estimator that fixes rank collapse in Fisher-weighted SVD, and LEMS, a layer-wise error surrogate solved with an integer linear program (ILP) for global rank allocation — and evaluates them on Mistral, Llama 3 and Qwen3 families from 4B up to 70B parameters.

What This Paper Is About

SVD-based compression shrinks each dense weight matrix into two smaller factors, but two problems limit it: existing methods allocate ranks across layers using proxies that are either poorly calibrated or blind to how errors compound with depth, and Fisher-weighted decomposition (which aligns truncation with loss curvature) collapses in effective rank because it is estimated from only a handful of per-sequence gradients. The paper's goal is to fix both — replacing the ad-hoc rank heuristic with a calibrated, depth-aware error model optimized globally, and replacing sequence-level Fisher estimation with token-level statistics.

Key Contributions

1. LEMS (Layer-wise Error Modeling Search). A decomposable per-layer cost model that multiplies three terms — a spectral "shape" curve from normalized tail energy, an empirically calibrated "scale" factor measured by KL divergence at a single reference point, and a depth-dependent "bias" that models error propagation — and then selects one rank configuration per layer globally by solving an ILP.
2. KFAC-SVD. A reformulation of Fisher-weighted SVD that estimates the Kronecker factors from token-wise second moments rather than per-sequence weight gradients, raising the effective sample count from k sequences to N ≈ S·T and removing the rank-k bottleneck.
3. Empirical validation across three model families. KFAC-SVD improves WikiText2 perplexity by an average of 15% over baselines; LEMS delivers accuracy gains of 1.7–4.8 p.p. over the best existing rank allocation methods and generalizes to activation-based SVD.
4. Scalability evidence. The method is shown on the Qwen3 family (4B–32B) across five compression ratios (0.5–0.9) and on Llama 3.1-70B, where LEMS reduces WikiText2 perplexity from 7.9 to 4.5 and raises accuracy from 62.6 to 66.6 (a 4.0 p.p. gain over uniform allocation).

Main Findings

• Rank allocation is the dominant bottleneck. Uniform allocation, ASVD, SVD-LLMv2 and MRCS frequently fail to beat uniform compression, which the authors attribute to noisy sensitivity proxies and constraints that force overly similar compression across heterogeneous layer types.
• LEMS beats all search baselines. Across Mistral-7B, Llama 3-8B and Qwen3-8B at ratios 0.8 and 0.6, LEMS achieves both the highest zero-shot accuracy and the lowest perplexity; the accuracy gain over the second-best method ranges from 1.68 to 4.79 p.p., and mean perplexity (averaged over WikiText2, C4 and PTB) drops by 39.34% to 83.65% relative to uniform allocation.
• ARS is the strongest baseline. The gradient-based ARS search is second-best in most settings, but the paper notes it requires gradients, more data and hyperparameter tuning to meet target ratios.
• ATP only partially helps. The one baseline that explicitly models error propagation (ATP) generally matches or narrowly beats uniform allocation, which the authors attribute to its assumption of linear error accumulation.
• KFAC-SVD fixes Fisher-based degradation. At a 0.7 ratio, KFAC-SVD improves WikiText2 perplexity by 38% over the best baseline (DOBI-SVD), rising to 65% when combined with LEMS; against prior Fisher-based methods specifically it wins by up to 7.2 p.p. without LEMS.
• GFWSVD collapses under identical conditions. The prior Fisher-based SVD baseline degrades severely (e.g., WikiText2 perplexity of 2569.75 on Llama 3-8B at 0.9 and 41798.89 at 0.7), consistent with the diagnosed conditioning/rank problem.
• Mild compression is nearly lossless with LEMS. At a 0.9 ratio, KFAC-SVD stays in the mid-50s in zero-shot accuracy while LEMS recovers 62.0% versus 63.3% for the original model on Llama 3-8B.
• One exception: Mistral-7B. Baselines such as ASVD reach higher zero-shot accuracy than KFAC-SVD despite worse perplexity, which the authors interpret as Mistral relying on features aligned with activation magnitude.
• Scaling results on Qwen3 (4B–32B, ratios 0.5–0.9). Averaged over ratios and models, LEMS cuts perplexity by 23% for KFAC-SVD and 34% for SVD-LLM, and raises zero-shot accuracy by +8.3 and +9.1 p.p. respectively; at ratio 0.5 it cuts perplexity by roughly 47% (KFAC-SVD) and roughly 57% (SVD-LLM) and recovers +6.3 / +8.7 p.p. It also removes the sharp accuracy dips uniform allocation produces on Qwen3-32B.
• Decomposition vs. allocation trade-off. KFAC-SVD gives roughly 15% lower perplexity than SVD-LLM under uniform allocation, but once LEMS is applied the two become tightly matched, with SVD-LLM keeping a marginal edge of about 0.1 p.p. in zero-shot accuracy — suggesting allocation, not decomposition, is the main source of loss under uniform compression.
• Ablation of LEMS components (Llama 3-8B / Qwen3-8B, WikiText2). Adding ILP search over spectral energy alone drops perplexity from 11.44 to 9.36 and 12.52 to 11.16 for about 1 minute of extra runtime (~18% and ~11% reductions); KL calibration brings this to 8.69 and 10.40 (~7% further) at a cost of 17–22 minutes that is cacheable across ratios; the cross-layer bias takes 2 more minutes and reaches 8.16 and 10.34, a 6% reduction for Llama 3 but marginal for Qwen3. At ratio 0.6 the bias term reduces Llama 3-8B perplexity by over 50% versus the version without it.
• Runtime is practical. LEMS takes 20–25 minutes for the initial run and 3 minutes for subsequent runs (cache reuse), measured on a single NVIDIA H200 GPU.

Methodology in Plain English

Fixing the decomposition (KFAC-SVD). Fisher-weighted SVD says: don't just minimize how much the weight matrix changes, minimize how much the loss changes, approximated as a quadratic form with a Fisher matrix F. Because F for an LLM layer is far too large, it is factored as a Kronecker product A ⊗ G, and the optimal low-rank solution is obtained by whitening the weight with both factors, truncating the SVD, and un-whitening — implemented as two consecutive linear layers with the bias kept unchanged. The problem was estimating A and G: the prior method fitted them from k full per-sequence gradients, so the estimate has rank at most k (e.g., k ≤ 256) while the parameter dimension d_out·d_in is orders of magnitude larger. The fix is to estimate A from the second moments of layer inputs and G from the second moments of output gradients, accumulated over individual tokens. With 256 sequences of length 2048 this gives roughly 5.2×10^5 effective samples, and the rank bounds become min(d_in, N) and min(d_out, N).

Fixing the allocation (LEMS). The authors frame allocation as three separate problems: granularity (you need a cost for every candidate rank of every layer, which is too expensive to measure by forward passes), sensitivity (spectral energy is not comparable across layer types), and propagation (layer errors are not additive; early errors amplify). Their surrogate cost for layer i at retained-parameter ratio c is a product of three terms:

• Shape: the normalized tail energy S_i(c), the ratio of discarded to total squared singular values of the whitened weight. This is cheap and gives a fine-grained curve, but only describes relative degradation within a layer.
• Scale: a single measured calibration point per layer — the KL divergence between the original and compressed model's output when that layer alone is compressed to a reference ratio c_ref (e.g., 0.1). Dividing by S_i(c_ref) yields a scalar α_i that rescales the spectral curve onto the loss scale.
• Bias: a soft depth penalty β(d_i) built from a harmonic tail sum over transformer block indices, with strength κ and decay γ tuned by lightweight Tree-structured Parzen Estimation. This assigns larger penalties to early blocks and decays with depth, without imposing rigid monotonicity.

Given all layer costs and a parameter budget, the method solves exactly one rank configuration per layer with an ILP — minimize the total surrogate cost subject to the summed retained parameters not exceeding the target — using the off-the-shelf Gurobi solver, which takes seconds. Search is restricted to per-layer retention in [0.3, 1.0].

Experimental setup. All methods share identical seeds, calibration data (256 random WikiText2 sequences of length 2048), and candidate ranks, differing only in rank selection. Data-dependent searches use 32 WikiText2 samples following ASVD. Evaluation covers WikiText2, PTB and C4 perplexity plus six zero-shot benchmarks (PIQA, OpenBookQA, HellaSwag, ARC-Easy/ARC-Challenge, Winogrande) via the LM-Eval library. Compression ratios refer to remaining parameters of linear layers, excluding the head.

Why This Matters

Impact on research. The paper separates two failure modes that prior work often conflated: bad rank allocation and bad decomposition. Its key diagnostic — that LEMS closes most of the gap between KFAC-SVD and the simpler activation-based SVD-LLM — reframes rank allocation as the primary lever, and its token-wise Fisher estimator is a general fix for a known numerical pathology rather than a task-specific trick. Formulating allocation as an ILP over a calibrated surrogate also connects LLM compression to the integer-programming literature already used in structured pruning.

Real-world applications:

• Deploying capable LLMs on consumer hardware, phones and embedded automotive compute where memory and bandwidth are the binding constraints.
• Serving many model replicas per accelerator in datacenter inference, where SVD's dense low-rank factors remain directly compatible with standard inference engines (unlike sparse or semi-structured pruning).
• Combining compression with quantization, which the paper notes is orthogonal to low-rank approximation, to stack savings on already-quantized deployments.
• On-device or edge assistants that need a single fixed model to fit a hard memory ceiling without retraining.

Industry relevance. The author list spans BMW Group alongside TU Munich, TU Wien and the University of Glasgow, and the paper is explicit about practical constraints: one H200 GPU, 20–25 minutes of one-time search cost, 3 minutes with caching, and no specialized sparse kernels required. That profile — post-training, minutes-scale, hardware-agnostic — is what makes it deployable rather than a research curiosity.

Future Directions

• Cross-token second moments. The token-wise Fisher approximation deliberately drops off-diagonal structure such as that induced by weight sharing; the authors flag this as a trade-off that could be revisited.
• Better propagation models. The depth bias is a heuristic harmonic profile tuned by TPE, and its gains were marginal on Qwen3 at ratio 0.8; a learned or architecture-specific propagation model is an open question, especially under aggressive compression where the effect grows largest.
• Closing the remaining gap at high compression. Even with LEMS, the recovered zero-shot accuracy on Llama 3-8B at 0.9 is 62.0% versus 63.3% for the original model, so lossless compression is not yet reached.
• Interaction with other low-rank families. The paper mentions supplement results on non-standard SVD variants (BasisSharing, BitStack) but leaves a systematic treatment of how LEMS's error model should be adapted to objectives that modify activations during the forward pass, such as DOBI-SVD.

Target Audience

Researchers and engineers working on post-training LLM compression, low-rank approximation, or second-order optimization for large networks; practitioners who need to hit a specific parameter or memory budget without retraining and want to know whether the effort is better spent on decomposition or allocation; and readers interested in the transfer of ILP-based global optimization from structured pruning to rank selection. Readers without background in SVD-based compression should expect a steep first section but a self-contained problem statement thereafter.

Authors’ abstract

Low-rank SVD-based compression offers a powerful strategy to reduce the computational costs of LLMs. However, existing methods face two key limitations: (i) global rank allocation, where uncalibrated error proxies fail to capture complex error propagation, and (ii) decomposition quality, where Fisher-based estimators suffer from severe rank collapse. In this work, we address these limitations by introducing Layer-wise Error Modeling Search (LEMS) and KFAC-SVD. LEMS advances rank allocation by introducing a layer-wise error surrogate that integrates local and global layer importance alongside a propagation bias, enabling effective global rank allocation via an ILP formulation. KFAC-SVD improves decomposition quality by utilizing token-wise statistics, mitigating the rank deficiency observed in prior Fisher-based SVD approaches. Across Mistral, Qwen3, and Llama 3 model families, we show that LEMS consistently outperforms existing search strategies, delivering significant zero-shot accuracy gains of up to 4.8 p.p. that generalize to model sizes of 70B parameters, while KFAC-SVD achieves an average perplexity improvement of 15%. Project page and code: https://lems-svd.github.io