1+1>2: A Synergistic Sparse and Low-Rank Compression Method for Large Language Models

AI summary

Overview

Research area: Efficient large language model deployment — specifically post-training compression (PTC), combining unstructured pruning with low-rank approximation.

Technical level: Advanced. The paper builds on Optimal Brain Surgeon (OBS) theory, Hessian-based salience metrics, and matrix decomposition, and frames compression as a constrained optimization problem.

Scope: The paper introduces SSLC (Synergistic Sparse and Low-Rank Compression), an iterative, data-aware method that jointly optimizes a sparse and a low-rank component of each weight matrix to compress LLaMA and Qwen2.5 models (7B–70B) with state-of-the-art results at 50% compression and no fine-tuning required.

What This Paper Is About

Large language models are expensive to store and serve because of their parameter count and memory bandwidth demands. Pruning (removing unimportant weights) and low-rank approximation (decomposing weight matrices into smaller matrices) each work reasonably well alone, but the paper argues their combination for LLMs has been underexplored. The goal is to decompose each weight matrix into a sparse part that keeps the discrete, full-rank, non-coherent information and a low-rank part that captures the shared, coherent structure — and to do it without retraining, so that compression does not break model quality.

Key Contributions

1. SSLC, a joint compression algorithm that integrates low-rank approximation with pruning into a single decomposition, mathematically retaining the orthogonality property of low-rank approximation and the full-rank preservation property of sparse reconstruction.

2. A unified data-aware optimization formulation. The paper formulates low-rank approximation and sparse optimization as one problem — minimizing ||(W − L − S)X||_F subject to rank(L) = r and sparsity(S) = k% — and solves it with an alternating iterative algorithm rather than treating pruning and SVD as separate stages.

3. A theoretical convergence argument. Drawing on OBS-style quadratic error analysis, the paper shows the reconstruction error decreases monotonically across alternating iterations, with Figure 5 showing more than 90% convergence within 40 iterations.

4. Empirical state-of-the-art results without fine-tuning, plus a parameter-efficient fine-tuning path: the low-rank factors U_t and V_t already produced by SSLC can be fine-tuned directly (with dimension D_low ≤ 128) instead of adding a separate LoRA branch, and the method is validated for speed on a simulated ViTCoD accelerator.

Main Findings

• Qwen2.5 compressed by 50% with no performance drop. The abstract states SSLC compresses Qwen2.5 by 50% with no performance drop and achieves at least a 1.63× speedup. In Table 2, Qwen2.5-7B scores 71.50 average zero-shot accuracy after fine-tuning versus 70.83 for the dense model (+0.67), and Qwen2.5-14B scores 74.19 versus 73.93 dense (+0.26).

• Better perplexity than standalone baselines at 50% remaining parameters without fine-tuning. On LLaMA1-7B, C4 perplexity is 8.91 for SSLC versus 9.31 (SparseGPT), 9.30 (Wanda), 9.13 (DSnoT), and 127.25 (SVD-LLM), against a dense value of 7.34. On WikiText-2 the same model gives 6.92 versus 5.68 dense, 7.22 SparseGPT, 7.24 Wanda, 7.15 DSnoT, and 24.52 SVD-LLM.

• Zero-shot accuracy advantage at 50% sparsity. Averaged over HellaSwag, Winogrande, BoolQ, PIQA, ARC-Easy, and ARC-Challenge, SSLC reaches 63.59 on LLaMA1-7B versus 66.31 dense, 63.12 SparseGPT, 62.77 Wanda, 63.91 DSnoT, and 39.07 SVD-LLM. The pattern holds across LLaMA2-7B, LLaMA3-8B, LLaMA1-13B, LLaMA2-13B, LLaMA3-70B, Qwen2.5-7B, and Qwen2.5-14B in Table 1.

• Compression "rate spread" from synergy. When retaining 80% of original weight salience, pure pruning needs to keep the top 42.3% of parameters (compressing 57.7%), whereas the synergistic method needs only a 32.3% sparse matrix plus 6.25% from the low-rank part, for 38.6% total retention — a 3.7% absolute reduction.

• Fine-tuned SSLC nearly closes the dense-model gap, without extra LoRA parameters. On LLaMA2-7B, SSLC averages 66.41 versus 66.96 dense (Δ −0.55), while SparseGPT with LoRA gets 65.22 (Δ −1.74) and Wanda with LoRA gets 65.39 (Δ −1.57). On LLaMA3-8B: 70.42 for SSLC (Δ −0.99) versus 69.50 SparseGPT (Δ −1.91) and 68.99 Wanda (Δ −2.42).

• A compressed model can beat a smaller dense model. Qwen2.5-14B compressed with SSLC (approximately 7B effective parameters) outperforms native dense Qwen2.5-7B on zero-shot tasks, with an average improvement of 1.1%.

• Speedups on the simulated ViTCoD accelerator at 50% compression. 1.74× (MHA) and 1.84× (FFN) for LLaMA2-7B; 1.63× (MHA) and 1.85× (FFN) for Qwen2.5-7B.

• Real-world throughput gains. Using nm-vLLM with 1024-token generation over 5 prompts, SSLC achieves throughput speedups of 1.34×–1.69× in the bandwidth bottleneck. Table 5 reports LLaMA2-7B throughput of 53.79 tokens/sec dense, 72.12 at 50%, 77.87 at 60%, and 89.87 at 70%; LLaMA1-7B gives 54.07, 73.02, 79.14, and 91.25 respectively.

• Ablation: iteration count. For LLaMA2-7B at 50% remaining parameters, average perplexity is 8.55 at T=0, 8.00 at T=10, 7.87 at T=20, 7.79 at T=30, 7.74 at T=40, 7.72 at T=50, and 7.71 at T=60. The authors select 40 iterations, since 40 iterations consume 33% less resources than 60 iterations.

• Ablation: rank. At a fixed 50% compression ratio on LLaMA2-7B, performance improves as rank increases from 32 to 128 and then decreases, so 128 is chosen. Pure pruning corresponds to rank=0 and pure low-rank to rank=1296. The provided text cuts off mid-sentence in this section.

• Sparsity robustness. Figure 6 shows SSLC consistently beating baselines at pruning intensities from 10% to 50% sparsity on LLaMA2-7B over WikiText-2, with the gap widening at higher sparsity; existing pruning methods also improve when wrapped in the SSLC framework.

• Overhead. Pruning completes in approximately 30 minutes for 7B models and about 1 hour for 14B models on standard hardware; the contribution bullet mentions roughly 3 GPU hours of pruning and fine-tuning for Qwen2.5-7B.

• Not reported in the provided content: the detailed per-dataset zero-shot tables referenced as Appendix D (Tables 10), and the remaining ablation details on random seed initialization and salience-based weight preservation strategy.

Methodology in Plain English

SSLC treats compression as a layer-by-layer reconstruction problem: for a layer's weight matrix W and its input activations X, it searches for a compressed replacement that keeps the layer's output as close as possible to the original, measuring error with the Frobenius norm.

The key move is splitting the weight matrix into two pieces at the same time — a low-rank piece L and a sparse piece S — instead of doing one and then the other. The low-rank piece uses singular value decomposition to find a small set of orthogonal bases that capture as much of the weight energy as possible; the sparse piece keeps the individual high-importance weights that a low-rank approximation would smear away. Because the ideal joint problem is NP-hard, the authors alternate: compute the sparse part given the current low-rank part, then compute the low-rank part given the current sparse part, and repeat for T iterations.

Two engineering choices make this practical. First, instead of a full SVD, they use randomized SVD via bilateral random projections with random matrices A_1 and A_2, which avoids the prohibitive cost of decomposing matrices of size 4096×4096 and 4096×11008. Second, they reserve the top 1% of weights by salience (the OBS-derived metric δ_ij = ΔW_ij²/[H⁻¹]_jj², with H ≈ XᵀX) and exclude those from the decomposition entirely; the remaining budget is allocated as (k−1)% to the sparse part and r × (m+n)/(m×n) to the low-rank part.

For recovery, rather than bolting on a separate LoRA adapter, they freeze the sparse matrix S_t and fine-tune only the U_t and V_t factors that the decomposition already produced. The calibration data is 128 sequences of 2048 tokens from the first slice of C4, the same setup used by Wanda, and fine-tuning uses the Stanford Alpaca dataset.

Why This Matters

Impact on research. The paper reframes pruning and low-rank approximation as two views of the same decomposition problem rather than competing techniques. It supplies a convergence argument showing the alternating scheme monotonically reduces the quadratic reconstruction error, and it demonstrates a fine-tuning path that reuses the compressed structure instead of adding parameters — a distinction from LoRA-based recovery schemes such as SparseGPT with LoRA. It also contrasts with prior sparse-plus-low-rank work for LLMs (LoSparse, LoRAshear, LoRAPrune, LoSA, LoRaP) that either require full-network training or lack joint optimization.

Real-world applications:

• Serving large models on bandwidth-constrained or consumer-grade hardware (the paper explicitly notes the method runs on consumer-grade graphics cards).
• Reducing deployment cost for long-context or high-throughput inference, where memory bandwidth rather than compute is the bottleneck.
• Producing compressed checkpoints that can be fine-tuned per downstream task using the existing low-rank factors.
• Deploying on accelerators with algorithm-hardware co-design for sparse-dense mixed workloads, as modeled by the ViTCoD simulator.

Industry relevance. The method needs no retraining for its headline results, finishes pruning in about 30 minutes for 7B models and about 1 hour for 14B models, and yields a compressed Qwen2.5-14B that beats dense Qwen2.5-7B on zero-shot tasks by 1.1% on average. Those properties map directly onto production inference economics, where a 50% parameter reduction with an at-least-1.63× speedup and no measured quality loss changes the cost curve for serving models. The authors are affiliated with Hikvision Research Institute, which signals deployment-oriented motivation.

Future Directions

• Layer-wise adaptive compression. The authors state that SSLC currently applies uniform compression ratios across all Transformer layers, which may underuse the differing sensitivities of layers; they propose exploring theoretically grounded layer-criticality metrics, potentially gradient-weighted Hessian analysis, to enable dynamic per-layer policies and Pareto-efficient accuracy/compute trade-offs.
• Closing the remaining fine-tuning gap. On LLaMA2-7B and LLaMA3-8B the fine-tuned SSLC still trails the dense model by 0.55 and 0.99 average points respectively, leaving room for recovery strategies beyond tuning the existing U_t and V_t factors.
• Extension beyond the tested families and tasks. The evaluation covers LLaMA-7B/13B, LLaMA2-7B/13B, LLaMA3-8B/70B, and Qwen2.5-7B/14B on language modeling perplexity and six zero-shot classification tasks; behavior on other architectures, reasoning-heavy benchmarks, or multimodal models is not reported.
• Combining with quantization. The paper focuses on pruning and low-rank approximation and cites quantization as a parallel post-training compression line; how SSLC stacks with quantization is not addressed in the provided content.

Target Audience

This paper suits researchers and engineers working on LLM efficiency — model compression, inference optimization, and hardware-aware deployment — as well as practitioners who need to shrink 7B-to-70B models for bandwidth- or memory-limited serving. It assumes familiarity with pruning terminology (salience, Hessian-based error, OBS), SVD, and the standard layer-wise reconstruction framing, so readers without that background will find the methodology sections dense. Readers interested in the systems side (accelerator simulation, throughput measurement) will find Section 5.5 and Appendix C most useful, while those interested in the optimization formulation should focus on Sections 3, 4.1, 4.2, and Appendix A.

Authors’ abstract

Large Language Models (LLMs) have demonstrated remarkable proficiency in language comprehension and generation; however, their widespread adoption is constrained by substantial bandwidth and computational demands. While pruning and low-rank approximation have each demonstrated promising performance individually, their synergy for LLMs remains underexplored. We introduce \underline{S}ynergistic \underline{S}parse and \underline{L}ow-Rank \underline{C}ompression (SSLC) methods for LLMs, which leverages the strengths of both techniques: low-rank approximation compresses the model by retaining its essential structure with minimal information loss, whereas sparse optimization eliminates non-essential weights, preserving those crucial for generalization. Based on theoretical analysis, we first formulate the low-rank approximation and sparse optimization as a unified problem and solve it by iterative optimization algorithm. Experiments on LLaMA and Qwen2.5 models (7B-70B) show that SSLC, without any additional training steps, consistently surpasses standalone methods, achieving state-of-the-arts results. Notably, SSLC compresses Qwen2.5 by 50\% with no performance drop and achieves at least 1.63$\times$ speedup, offering a practical solution for efficient LLM deployment.