SkipCat: Rank-Maximized Low-Rank Compression of Large Language Models via Shared Projection and Block Skipping

AI summary

Overview

Research area: Efficient inference for large language models (LLMs) — specifically low-rank (SVD-based) weight compression for deployment on resource-constrained hardware. Published in the Natural Language Processing category (arXiv:2512.13494v1 [cs.CL], 15 December 2025) by authors from National Yang Ming Chiao Tung University, Macronix International Co., Ltd., Skymizer Taiwan Inc., and The University of Texas at Austin.

Technical level: Intermediate. The paper assumes familiarity with matrix factorization (SVD), transformer layer structure (attention and MLP modules), and standard LLM compression terminology, but each technique is explained from its mathematical foundations.

Scope: The paper proposes and evaluates SkipCat, a training-free low-rank compression framework combining intra-layer shared projections (Cat) and Schur-complement-based block skipping (Skip), benchmarking it against ASVD, Basis Sharing, Dobi-SVD, and SVD-LLM on LLaMA2-7B/13B and Qwen3-8B/14B.

What This Paper Is About

Low-rank compression replaces a large weight matrix with two smaller matrices, but a naïve SVD decomposition only starts saving computation and memory when the retained rank drops below roughly half of the full rank (the constraint is written as r < d_in·d_out/(d_in+d_out); for a full-rank square matrix, r < R/2). Cutting the rank that aggressively damages accuracy. This paper asks whether a model can keep far more ranks — and therefore far more of its original capability — while still hitting the same compression rate. SkipCat answers this with two structural changes to how the low-rank factors are built and multiplied.

Key Contributions

1. Cat — Intra-Layer Shared Low-Rank Projection using Matrix Concatenation. Matrices inside a layer that consume the same input (q, k, v in attention; gate and up in the MLP) are concatenated along the output dimension and factorized jointly, so they share a single projection matrix. With C concatenated matrices, the amortized parameter count per matrix falls to r(d_in + C·d_out)/C and the amortized FLOPs to 2r(d_in + C·d_out)/C, freeing budget to retain more ranks.

2. Skip — Block Skipping via Schur Complement. The low-rank projection matrix A is partitioned into A_1 ∈ R^{r×r} and A_2 ∈ R^{r×(d_in−r)}, and A_1 is absorbed into the reconstruction matrix, so the computation becomes B′(x_1 + A′x_2) with B′ = BA_1 and A′ = A_1^{-1}A_2. This drops parameter count to r(d_in + d_out − r) and FLOPs to 2r(d_in + d_out − r) + r.

3. A numerical-stability fix for low-precision inference. Because an ill-conditioned A_1 makes A_1^{-1} large and causes FP16 overflow, the authors apply a column permutation to A, identified via Strong Rank-Revealing QR factorization, so the leading r columns form a well-conditioned submatrix. The paper reports the resulting activations are nearly two orders of magnitude smaller and more uniformly distributed.

4. Empirical validation across model families and compression levels, including a training-free comparison against four prior methods, an ablation isolating Cat and Skip, a quantization-compatibility test, and a LoRA fine-tuning study — plus extension to LLaMA2-13B and Qwen3-14B.

Main Findings

• Training-free accuracy gain: Without any additional fine-tuning, SkipCat outperforms previous low-rank compression approaches by up to a 7% accuracy improvement on zero-shot tasks under the same compression rate (stated in the abstract; on LLaMA2-7B at 30% compression the paper reports a relative improvement of 7% versus the best competitor's drop).

• LLaMA2-7B, 30% compression: SkipCat reaches 7.65 WikiText-2 perplexity and 11.57 C4 perplexity with 48.46% average zero-shot accuracy (a 6.34% drop from the dense model's 54.79%). The best competing method, SVD-LLM, reaches 11.75 / 19.37 perplexity and 41.25% average accuracy (13.54% drop). ASVD degrades to 208.55 WikiText-2 perplexity and 33.12% average accuracy.

• SkipCat at higher compression beats rivals at lower compression: On LLaMA2-7B, SkipCat at 30% compression achieves lower perplexity than every other method compressed at only 20%. Existing methods at 30% inflate WikiText-2 perplexity by 2.1× relative to the uncompressed model, whereas SkipCat incurs only a 1.4× increase.

• LLaMA2-7B, 20% compression: SkipCat scores 6.29 / 8.95 perplexity and 52.59% average accuracy — a 2.20% drop. SVD-LLM scores 8.82 / 13.42 and 44.84% (9.95% drop); Basis Sharing 9.39 / 23.30 and 43.77% (11.02% drop); Dobi-SVD 9.39 / 19.46 and 40.83% (13.97% drop); ASVD 9.06 / 11.66 and 48.81% (5.98% drop).

• Qwen3-8B results track the same pattern: Dense scores 9.72 / 15.42 perplexity and 60.28% average accuracy. At 20% compression SkipCat reaches 11.68 / 19.09 and 56.42% (3.86% drop) versus SVD-LLM's 14.33 / 23.21 and 51.66% (8.62% drop). At 30% compression SkipCat reaches 13.81 / 23.70 and 51.71% (8.57% drop) versus SVD-LLM's 19.17 / 32.69 and 44.35% (15.93% drop).

• Scaling to larger models holds up: On LLaMA2-13B at 20% compression, SkipCat averages 54.15% versus SVD-LLM's 50.09% (dense: 58.07%); at 30%, 49.29% versus 46.15%. On Qwen3-14B at 20%, SkipCat averages 60.15% versus SVD-LLM's 55.11% (dense: 64.01%); at 30%, 56.04% versus 50.19%.

• Ablation confirms both techniques contribute: On LLaMA2-7B at 20% compression, naïve SVD alone gives 8.82 WikiText-2 / 13.42 C4 perplexity. Cat only: 7.84 / 11.99. Skip only: 6.71 / 9.32. Both together: 6.29 / 8.95. Adding HQQ 8-bit quantization on top yields 6.29 / 8.96 — essentially no additional perplexity loss.

• Fine-tuning amplifies the advantage: After LoRA fine-tuning on Alpaca with LLaMA2-7B, SkipCat at 20% compression averages 54.41% (0.39% drop from the 54.79% dense baseline) versus SVD-LLM's 51.02% (3.78% drop). At 40%: 48.65% versus 46.91%. At 60%: 41.16% versus 39.50%. At 80%: 32.82% versus 32.33% (drops of 21.97% and 22.47% respectively, where the gap narrows).

• Diminishing advantage at extreme compression: The paper states that the benefit of preserving more ranks diminishes under higher compression rates, and that combining Skip with Cat (red curve in Figure 1) is what keeps SkipCat inside the effective compression region at those levels.

Methodology in Plain English

The researchers start from the arithmetic of low-rank compression: a weight matrix W is approximated as B·A, where A projects the input into a lower-dimensional space of rank r and B reconstructs the output. Counting parameters and multiply-adds reveals that this only wins when r falls below d_in·d_out/(d_in+d_out).

Their first move exploits a structural redundancy in transformers. Inside one attention layer, the query, key, and value weight matrices all read the same input vector; inside the MLP, the gate and up matrices do too. Instead of factorizing each separately, they stack these matrices along the output dimension into one taller matrix and factorize the stack. The resulting single A matrix is shared, so each individual matrix no longer needs its own projection — the saved parameters are spent on a higher rank r.

Their second move borrows a trick from the Schur complement. Splitting A into a square block A_1 and a rectangular block A_2 lets them algebraically absorb A_1 into B, producing a computation B′(x_1 + A′x_2) that skips the sub-block explicitly involving A_1. This reduces both parameter storage and FLOPs, again freeing budget for a larger r.

The catch is numerical: if A_1 is ill-conditioned, its inverse blows up, A′ becomes huge, and activations overflow FP16 — the paper's Figure 3 shows such outliers. Their fix is to permute the columns of A before factorizing, using Strong Rank-Revealing QR to pick a well-conditioned leading block. Both techniques are then combined, with block skipping applied to every low-rank projection.

Evaluation is performed on LLaMA2 (7B and 13B) and Qwen3 (8B and 14B). Perplexity is measured on WikiText-2 and C4; zero-shot accuracy is measured with LM-Evaluation-Harness on ARC-Easy, ARC-Challenge, HellaSwag, OpenBookQA, WinoGrande, MathQA, and PIQA. Compression rates are defined over the memory transfer cost of the core weight matrices (q, k, v, o, gate, up, down). SkipCat and SVD-LLM both use whitening on samples from the WikiText-2 and C4 training splits. Baselines are run from official codebases; Basis Sharing's compression rate is recalculated under the authors' own standard. LoRA fine-tuning follows the SVD-LLM configuration on the Alpaca dataset.

Why This Matters

Impact on research. The paper reframes low-rank compression as a rank-maximization problem rather than a rank-truncation problem: instead of asking how few ranks a model can survive with, it asks how many ranks fit inside a fixed budget once shared structure and skipped sub-blocks are accounted for. The numerical-stability analysis also connects low-rank compression to a practical FP16 deployment issue that is easy to overlook in simulation-only studies.

Real-world applications:

• Running LLMs on edge devices with limited memory capacity, where the paper notes parameter counts can exceed available memory and high computational cost causes latency and energy penalties.
• Intelligent home devices, one of the deployment scenarios the introduction names alongside professional problem-solving.
• Domain-specific adaptation, where the paper argues additional fine-tuning can both improve downstream task accuracy and enable higher compression rates.
• Stacking with quantization — the ablation shows 8-bit HQQ quantization added on top of SkipCat incurs essentially no further perplexity cost, an attractive combination for multi-level memory savings.

Industry relevance. Two of the author affiliations are companies (Macronix International, Skymizer Taiwan), suggesting direct interest in deployable compression for hardware-constrained inference. All headline results are in the training-free setting, meaning the compression can be applied to an existing checkpoint without a retraining pipeline.

Future Directions

• Closing the gap at extreme compression. The paper explicitly notes the advantage of preserving more ranks diminishes under higher compression rates; the fine-tuning results at 80% compression show SkipCat at 32.82% versus SVD-LLM at 32.33%, a much narrower margin than at 20%. Understanding and extending performance in this regime is an open question.

• Hardware realization of block skipping. The saved FLOPs and parameter counts are theoretical; the paper reports no measured wall-clock latency, throughput, or energy figures on actual edge hardware. Translating block skipping into kernel-level or hardware-level speedups remains to be demonstrated.

• Broader model and architecture coverage. Only LLaMA2-7B/13B and Qwen3-8B/14B are evaluated. The paper frames its findings as opening new possibilities for efficient model compression generally, which invites testing on other architectures and modalities.

• Deeper integration with quantization and fine-tuning. The HQQ 8-bit result and the LoRA fine-tuning study each show one configuration; the paper proposes combining parameter-level compression with precision-level quantization as a path to greater compression, which leaves room for systematic exploration of joint compression and adaptation pipelines.

Target Audience

This paper is most useful to machine learning systems researchers and engineers working on LLM efficiency, particularly those implementing or benchmarking low-rank and SVD-based compression. Practitioners deploying models to edge or memory-constrained hardware will find the training-free results and the FP16 stability analysis directly actionable. Graduate students studying model compression will benefit from the paper's clear derivation of when low-rank compression actually pays off (the r < d_in·d_out/(d_in+d_out) condition) and its Appendix A treatment of block skipping as a Schur complement. Readers seeking measured latency, energy consumption, or on-device throughput numbers will not find them in this paper.

Authors’ abstract

Large language models (LLM) have achieved remarkable performance across a wide range of tasks. However, their substantial parameter sizes pose significant challenges for deployment on edge devices with limited computational and memory resources. Low-rank compression is a promising approach to address this issue, as it reduces both computational and memory costs, making LLM more suitable for resource-constrained environments. Nonetheless, naïve low-rank compression methods require a significant reduction in the retained rank to achieve meaningful memory and computation savings. For a low-rank model, the ranks need to be reduced by more than half to yield efficiency gains. Such aggressive truncation, however, typically results in substantial performance degradation. To address this trade-off, we propose SkipCat, a novel low-rank compression framework that enables the use of higher ranks while achieving the same compression rates. First, we introduce an intra-layer shared low-rank projection method, where multiple matrices that share the same input use a common projection. This reduces redundancy and improves compression efficiency. Second, we propose a block skipping technique that omits computations and memory transfers for selected sub-blocks within the low-rank decomposition. These two techniques jointly enable our compressed model to retain more effective ranks under the same compression budget. Experimental results show that, without any additional fine-tuning, our method outperforms previous low-rank compression approaches by 7% accuracy improvement on zero-shot tasks under the same compression rate. These results highlight the effectiveness of our rank-maximized compression strategy in preserving model performance under tight resource constraints.