FLRC: Fine-grained Low-Rank Compressor for Efficient LLM Inference

AI summary

Overview

Research area: Efficient large language model (LLM) inference, specifically low-rank (SVD-based) model compression for deployment on resource-constrained hardware.

Technical level: Intermediate. The paper assumes familiarity with transformer architectures, singular value decomposition, low-rank weight compression, and the distinction between the prefill and decoding stages of LLM inference.

Scope: This paper introduces FLRC (Fine-grained Low-Rank Compressor), a two-part framework that assigns a different rank budget to every projection in every layer and then reduces the active rank progressively as tokens are generated, targeting generation tasks where prior low-rank compression methods degrade sharply.

What This Paper Is About

Low-rank compression can shrink LLM memory and compute, but applying one uniform compression ratio to every layer damages accuracy, and existing methods also perform poorly on generation tasks that require many decoding steps, such as summarization. The paper's goal is to allocate an optimal rank to each individual projection (not just each layer) quickly, and to keep text generation quality high while still achieving a large overall compression rate.

Key Contributions

1. Fisher-based Layer-wise Rank Allocation (FLRA): a single-pass, gradient-based algorithm that scores every projection in every layer using the squared product of its weights and gradients, then distributes a global rank budget proportionally to those scores.
2. Progressive Low-Rank Decoding (PLRD): a dynamic decoding scheme in which the model's total active rank is non-increasing across the token sequence, so early tokens use more parameters and later tokens use fewer, at the same overall average compression rate.
3. A scheduler that uses a calibration dataset to pick, for a target overall rank budget, the point(s) during generation at which the model switches to a higher compression rate.
4. Empirical validation across multiple LLaMA-family models (3B, 7B, 8B, 13B, and 30B) and benchmarks, showing gains on generation tasks (DialogSum, CNN/DM) and understanding tasks (Wikitext2 perplexity plus seven zero-shot tasks), along with a measured 49-fold reduction in rank-search time and end-to-end and offloading throughput speedups.

Main Findings

• Generation results are the headline achievement. On Llama-3-8B-Instruct at a 20% compression rate on DialogSum, FLRC reaches 17.35 ROUGE-L versus 0.24 for SVD-LLM and 0.10 for ASVD; BERTScore is 86.00 versus 78.12 (SVD-LLM), 80.07 (ASVD), and an 86.79 uncompressed baseline.
• The gap persists at higher compression. At 30% compression on Llama-3-8B-Instruct, FLRC scores 8.09 ROUGE-L on DialogSum versus 0.41 (SVD-LLM) and 0.53 (ASVD); on CNN/DM it scores 10.83 versus 3.98 (SVD-LLM) and 0.07 (ASVD).
• Gains are smaller but consistent on Llama-2-7B-Chat. At 20% compression FLRC records 17.22 ROUGE-L on DialogSum (vs. 15.44 ASVD, 13.62 SVD-LLM) and 19.84 on CNN/DM (vs. 7.94 ASVD, 19.71 SVD-LLM); at 30% it reaches 14.91 and 17.28, versus 6.47/3.44 (ASVD) and 2.34/15.56 (SVD-LLM).
• Understanding tasks improve too. On Llama-3-8B, FLRC reaches 12.53 Wikitext2 perplexity at 20% compression (SVD-LLM 14.72, ASVD 3206.80, baseline 6.14) and 25.46 at 30% (SVD-LLM 33.13, ASVD 28566.03). Average zero-shot accuracy is 43.66 at 20% and 35.76 at 30%, versus 41.63/34.88 for SVD-LLM.
• Rank search is far faster. On an A100 GPU, ASVD's perplexity-based search takes approximately 147 minutes; FLRC's takes 3 minutes, a 49-fold speedup.
• Better allocation, not just faster. At 20% compression, FLRC achieves 12.53 Wikitext2 perplexity versus ASVD's 22.69; at 30%, 25.46 versus 128.96. Against other allocation strategies on Llama-2-7B-Chat DialogSum, FLRC scores 17.22 at 20% (PrunerGPT 16.28, MoDeGPT 3.91) and 14.91 at 30% (PrunerGPT 10.81, MoDeGPT 2.43).
• The decoding schedule matters. Holding the average parameter count equal on DialogSum, decreased-rank decoding scores 19.87 ROUGE-L, static rank decoding 14.71, and increased-rank decoding 8.59.
• Both components contribute. In an ablation at 20% compression on Llama-3-8B-Instruct, SVD-LLM alone gives 0.24 ROUGE-L, adding FLRA raises it to 13.28, and adding both FLRA and PLRD yields 17.35.
• The importance metric choice is validated. Combining gradient and weight (20.23 ROUGE-L at 10% compression, 17.35 at 20%) beats weight-only (0.44 and 0.07) and gradient-only (15.74 and 2.16) on Llama-3-8B-Instruct DialogSum.
• Generalization across model sizes. On Llama3.2-3B DialogSum, FLRC scores 13.98 at 10% and 9.94 at 20% versus SVD-LLM's 7.09 and 3.55. On Llama-2-13B, FLRC scores 17.99 at 10% and 17.43 at 20% versus SVD-LLM's 16.94 and 0.18. On a 30B model, FLRC holds 18.95 ROUGE-L at 20%, 18.98 at 30%, and 18.19 at 40%, where SVD-LLM collapses to 0.00 ROUGE-L at 40%.
• Throughput gains depend on the bottleneck. With batch size 512, sequence length 32, and 128 generated tokens on Llama-3-8B-Instruct, throughput rises from 3646.62 tokens/sec (baseline) to 3856.99 (20%, 1.06x), 4051.53 (30%, 1.11x), and 5290.33 (40%, 1.45x). In an offloading setup capped at roughly 8GB VRAM with batch size 1, it rises from 1.20 tokens/sec to 1.40 (1.17x), 1.83 (1.53x), and 2.54 (2.12x).
• Low precision composes with the method. At INT8, DialogSum ROUGE-L is 17.48 (Llama-3-8B-Instruct, 20%) and 17.47 (Llama-2-7B-Chat, 20%), versus 17.35 and 17.22 in FP16.

Methodology in Plain English

The framework has two stages.

Stage one — deciding how much to compress each part. The authors run a small calibration set through the model and back-propagate to get gradients. For every projection matrix (q, k, v, o, gate, up, and down projections) in every layer, they multiply each weight by its gradient, square the products, and sum them into a single importance score. Squaring the product means a component only scores highly if both its weights and its gradients are large. Summing all scores across the model gives a total, and each projection then receives a share of the total rank budget equal to its share of the importance. This is a single pass with no iterative search, unlike Bayesian-optimization or Hessian-based approaches.

Stage two — using fewer ranks as generation proceeds. Because SVD orders the resulting channels by importance, a compressed projection can be truncated simply by keeping the top k rows or columns. FLRC reduces k over the course of generation according to a schedule chosen ahead of time from a calibration set. The rank budget for token t is never larger than for token t−1, so early tokens — which the authors argue shape output coherence most — get more parameters, and later tokens get fewer.

The pipeline follows SVD-LLM: weights are decomposed with SVD-LLM's truncation-aware data whitening, then FLRA and PLRD are applied. The authors deliberately omit SVD-LLM's weight-updating (PEFT fine-tuning) steps to keep the comparison fair. Calibration for rank allocation uses 256 sequences of length 2048 sampled from the Wikitext2 training set; the scheduler uses 500 samples from the DialogSum training set. Compression rate is defined as the average percentage of parameters omitted per token, computed across both the prefilling and decoding stages.

Why This Matters

Impact on research. The paper reframes low-rank compression evaluation: prior work largely reports perplexity or single-token common-sense accuracy, which the authors show hides catastrophic failure during multi-step generation. It also demonstrates that dynamic, token-varying rank is a viable alternative to static compressed models, and that a Fisher-style single-pass criterion can be both faster and more accurate than iterative search.

Real-world applications:

• On-device summarization and document briefing on phones or laptops with limited memory.
• Edge servers and single-user inference where model weights must be offloaded to host DRAM and streamed to GPU memory.
• Serving compressed LLMs under high-throughput, compute-bound batch workloads, where the measured speedup scales with compression rate.
• Low-precision deployment stacks, since INT8 combined with the method showed no accuracy drop in the reported tests.

Industry relevance. The compression rate as defined here maps directly to memory footprint and parameter movement, which are the binding constraints for on-premise and edge inference. The 49-fold reduction in search time shortens the tuning cycle needed before deployment, and the offloading results (up to 2.12x) target exactly the memory-bound regime that vendors of consumer GPUs and edge accelerators face.

Future Directions

• Scheduler overhead reduction. The authors explicitly flag that optimizing the scheduler for dynamic rank allocation remains a challenge, since switching ranks introduces overhead; they plan engineering optimizations and kernel design to reduce it.
• Making scheduling itself cheaper and more robust. The scheduler is currently selected from a calibration set using a metric such as BERTScore; how sensitive the chosen schedule is to that set and metric is left open.
• Calibration-data dependence. The authors report that models score better on data resembling their calibration set (e.g., Wikitext2 calibration helps Wikitext2 perplexity but hurts C4), which raises the question of how to select calibration data for unknown deployment distributions.
• Scaling the observed trend. FLRC showed a smaller accuracy drop on the 30B model than on smaller ones, suggesting compression efficiency improves with model size — whether this continues to much larger models is untested here.

Target Audience

Researchers and engineers working on LLM inference efficiency, model compression, and edge or on-device deployment will get the most from this paper, particularly those already familiar with SVD-based compression methods such as ASVD and SVD-LLM. Practitioners deciding whether low-rank compression is viable for generation workloads, and anyone benchmarking compression methods on summarization rather than perplexity alone, are the secondary audience.

Authors’ abstract

Although large language models (LLM) have achieved remarkable performance, their enormous parameter counts hinder deployment on resource-constrained hardware. Low-rank compression can reduce both memory usage and computational demand, but applying a uniform compression ratio across all layers often leads to significant performance degradation, and previous methods perform poorly during decoding. To address these issues, we propose the Fine-grained Low-Rank Compressor (FLRC), which efficiently determines an optimal rank allocation for each layer, and incorporates progressive low-rank decoding to maintain text generation quality. Comprehensive experiments on diverse benchmarks demonstrate the superiority of FLRC, achieving up to a 17% improvement in ROUGE-L on summarization tasks compared to state-of-the-art low-rank compression methods, establishing a more robust and efficient framework to improve LLM inference.