UniComp: A Unified Evaluation of Large Language Model Compression via Pruning, Quantization, and Distillation

AI summary

Overview

Research area: Efficient machine learning / large language model compression and evaluation methodology.

Technical level: Intermediate. The paper assumes familiarity with LLMs, pruning, quantization, and knowledge distillation, but its findings are presented at a level accessible to anyone who works with model deployment.

Scope: This paper introduces UniComp, a unified benchmark framework that compares pruning, quantization, and knowledge distillation across performance, reliability, and efficiency, evaluating seven compression techniques on over 40 datasets with LLaMA-3.1-8B and Qwen-2.5-7B as the main subjects.

What This Paper Is About

Existing comparisons of LLM compression methods mostly test pruning and quantization on knowledge-heavy multiple-choice benchmarks, which leaves open the question of what compression does to reasoning, multilingual ability, instruction following, and safety-related behavior. UniComp addresses this by evaluating compressed models along three dimensions — performance, reliability, and efficiency — using capability-oriented and safety-oriented benchmarks plus a hardware-aware efficiency analysis, and by treating knowledge distillation as a compression technique on equal footing with pruning and quantization.

Key Contributions

1. A unified evaluation framework. UniComp compares pruning, quantization, and knowledge distillation across performance, reliability, and efficiency, using 13 metrics over more than 40 datasets, code available at jvonrad/UniComp.

2. Evidence of a knowledge bias. Compression largely preserves factual recall while degrading multi-step reasoning, multilingual and cultural understanding, and instruction following disproportionately. Quantization is described as a partial exception.

3. Evidence of a performance-reliability decoupling. Retained capability scores do not predict preserved safety, fairness, or privacy, so performance retention should not be used as a proxy for reliability.

4. A calibration intervention. A reasoning-aware calibration set built from MATH, GSM8K, and ARC-c yields up to a 50% relative improvement in reasoning performance for pruned models.

Main Findings

• Knowledge is the most robust dimension. On LLaMA-3.1-8B under Wanda pruning at 50% sparsity, knowledge retention is 83.66% while reasoning retention is 40.15%; Minitron-Depth drops from 92.25% knowledge retention to 46.96% reasoning retention. The same pattern is reported for LLaMA-3.2-3B and Qwen-2.5-3B.

• Reasoning degrades most sharply. For LLaMA-3.1-8B, Wanda pruning reduces GSM8K from 76.80 to 19.48 and MATH-500 from 30.20 to 7.60. GPQA-Diamond is comparatively more robust, which the authors attribute to its multiple-choice format. Minitron-Depth, produced by removing the latter half of the transformer layers, performs substantially worse than Minitron-Width despite the same distillation procedure; a layer-wise ablation suggests this reflects depth reduction itself rather than the location of the removed layers.

• Quantization consistently outperforms other paradigms on knowledge. All three quantization methods achieve almost full knowledge retention on both models (S_K > 97). Even so, quantization is not a drop-in replacement: on LLaMA, GPTQ reaches S_R = 81.20, indicating that small quantization errors may accumulate over long reasoning chains. Weight-only methods (AWQ, GPTQ at W4A16) and weight-activation quantization (SmoothQuant at W8A8) show broadly comparable performance.

• Soft pruning helps distillation. Low-Rank Clone, which combines soft pruning with distillation, substantially outperforms the Minitron counterpart on multilingual performance and instruction following despite being distilled on an order of magnitude fewer tokens (20B vs. 200B). It remains limited in advanced reasoning (S_R = 57.33).

• Multilingual performance degrades disproportionately. On LLaMA, GPTQ shows S_K = 99.92 versus S_Mul = 90.10, and Wanda shows S_K = 83.66 versus S_Mul = 68.20. This difference is more pronounced in LLaMA-3.1-8B than in Qwen-2.5-7B. Analysis between high- and low-resource languages revealed no consistent additional degradation on low-resource languages.

• Performance and reliability are decoupled. Across all 17 method–model pairs, truthfulness, robustness, and ethics correlate strongly with retained performance (ρ = 0.73 to 0.90, p ≤ 0.05), while safety (ρ = -0.04), fairness (ρ = +0.36), and privacy (ρ = +0.33) show no significant correlation.

• Reliability preservation is model- and metric-dependent. Robustness, safety, and ethics are the most consistently preserved (S_ROB > 73, S_ETH > 72), while truthfulness and fairness show higher variance. Quantization is otherwise strong, with the notable exception of GPTQ, whose safety drops on both models. Compared with LLaMA, Qwen shows substantially smaller reliability degradation across all dimensions except safety; LLaMA's safety remains notably robust (S_SAFE > 84).

• Distillation preserves reliability less predictably. On LLaMA-3.1-8B, both Minitron variants underperform across most reliability dimensions. Low-Rank Clone on Qwen-2.5-7B achieves the strongest fairness score but remains weak on safety.

• Quantization gives the best overall efficiency trade-off. On LLaMA-3.1-8B, AWQ at W4A16 reports 38,150 t/s throughput, 661 ms latency, 10.53 GB inference GPU memory, 5.40 GB model size, 81.2 inference efficiency score, 16 m 32 s compression time, 42.1 GB peak compression memory, and a compute cost score of 12.53. GPTQ reports 41,562 t/s, 730 ms, 10.53 GB, 5.40 GB, 81.2, 10 m, 16 GB, and 26.14.

• Distillation maximizes efficiency at high compute cost. Minitron-Depth reaches the maximum runtime acceleration score of 100.0 at 78,606 t/s and 548 ms latency, with an inference efficiency score of 78.7, but its estimated training time is 140 hours on 20,480 GB of memory, giving a compute cost score of 0.01.

• Pruning benefits are limited to specialized settings. Meaningful runtime acceleration appears only under semi-structured 2:4 sparsity — Wanda at 2:4 reaches 71,194 t/s and 588 ms versus 41,498 t/s and 955 ms for the baseline — because hardware support can be leveraged there. Unstructured pruning shows weaker gains despite low compute cost; Wanda has the minimal compute cost, at 41 s compression time.

• Scale and architecture matter. Larger models are generally more robust to compression. Smaller models remain highly sensitive in reasoning: pruning reduces retained performance to as low as 6–25% for 0.6B and 1.7B models, while quantization can still incur losses of up to 30%. Pruning favors MoE models but struggles on reasoning, whereas quantization shows the opposite trend.

• Calibration data matters for pruning but not quantization. The default C4-based calibration data used by SparseGPT and Wanda is mismatched with reasoning tasks. With a reasoning-centric calibration set from MATH, GSM8K, and ARC-c, GSM8K accuracy on pruned LLaMA-3.1-8B improves from 36.9% to 55% (+50% relative) while knowledge performance is preserved. The same strategy produced no significant improvements for quantized models.

Methodology in Plain English

The authors built a single measurement harness rather than running many separate studies. They selected seven representative compression techniques: two pruning methods (SparseGPT and Wanda, removing 50% of parameters through unstructured and semi-structured 2:4 pruning), three quantization methods (GPTQ and AWQ at weight-only W4A16, SmoothQuant at weight-activation W8A8), and two distillation methods (Minitron, using the open-source LLaMA-3.1-Minitron-4B 50% depth and width pruned variants, and Low-Rank Clone, using the open-source LRC-4B model distilled from Qwen-2.5-7B). To keep the comparison fair, distillation is included only where the student is derived directly from the teacher without a separately pretrained model.

Main experiments use LLaMA-3.1-8B and Qwen-2.5-7B. For generalization, the authors also cover LLaMA-2 (7B, 13B, 70B), LLaMA-3.1 (8B, 70B), Qwen-3 (0.6B, 1.7B, 4B, 8B, 14B, 32B), DeepSeek-R1 (Distill-LLaMA-8B, Distill-LLaMA-70B), and the MoE model Qwen-3-30B-A3. Instruction-tuned models are used unless otherwise stated.

Performance is measured with MMLU and ARC-E/C for academic knowledge; HellaSwag, PIQA, and Winogrande for commonsense; GSM8K (4-shot), MATH-500 (4-shot), and GPQA-Diamond (5-shot) for reasoning; Global-MMLU-Lite across 14 languages and BBQ for multilingual and bias-related ability; and IFBench for instruction following. Reliability follows the TrustLLM protocol across truthfulness, safety, fairness, robustness, privacy, and ethics, drawing on 30 datasets including ConfAIde, MoralChoice, HaluEval, StereoSet, and Do-Not-Answer. Twelve of those 30 benchmarks are judged by GPT-4o and a Longformer-based harmful-response classifier; validating both against human annotation on 200 stratified instances showed substantial-to-near-perfect agreement (GPT-4o: 97%, κ = 0.957; Longformer: 91%, κ = 0.714).

Scoring uses a simple retention ratio: performance and reliability scores are normalized against the base model, where 100 means full retention. Efficiency metrics are different — because they are inherently comparative, each efficiency metric is normalized against the best-performing method and combined via a geometric mean to penalize bottlenecks. Runtime acceleration combines throughput and latency; inference efficiency combines GPU memory, model size on disk, and theoretical FLOPs; compute cost combines total compression time and peak GPU memory.

Experiments ran on Nvidia H100 GPUs using Lighteval and lm-evaluation-harness with vLLM, with compression applied through vLLM's llm-compressor framework with default hyperparameters. Decoding is deterministic: greedy decoding with temperature 0.0, no nucleus or top-k sampling, and a cap of 8,192 generated tokens for GSM8K and MATH-500. Throughput and latency use vLLM version 0.11.3 with an input length of 1,024 tokens and output length of 16 tokens. Memory footprint is measured via WikiText perplexity at batch size 1 and sequence length 4,096, reporting peak GPU memory. The order in which the paper presents findings is performance first, then reliability, then efficiency.

Why This Matters

Impact on research: the paper argues that the compression field has implicitly optimized for MMLU- and ARC-style retention, a Goodhart's Law effect ("When a measure becomes a target, it ceases to be a good measure"), which may explain why knowledge is uniformly well preserved while reasoning, which was not optimized for, is not. It makes the case that reliability should be an explicit evaluation objective rather than inferred from capability scores.

Real-world applications:

• On-device or edge deployment of compressed models, where compression is a constraint rather than a choice and re-alignment after compression is often unavailable.
• Privacy-sensitive deployments, where the finding that privacy retention does not track performance retention means unmeasured leakage risk.
• Safety-critical deployments, where jailbreak resistance can silently degrade even when a model passes every capability benchmark.
• Resource-constrained serving, where the alternative efficiency profiles of quantization (balanced trade-off) and distillation (peak efficiency at high compute cost) inform method selection.

Industry relevance: practitioners conventionally validate compression on knowledge-centric benchmarks, and this paper shows that choice can mask degradation in reasoning, multilingual performance, instruction following, safety, fairness, and privacy. The efficiency tables give concrete guidance on when quantization, 2:4 pruning, or distillation is worth its overhead, and the calibration result offers a zero-cost improvement lever at compression time.

Future Directions

• Broaden the evaluation scope. The main evaluation covers only LLaMA-3.1-8B and Qwen-2.5-7B, and the authors identify this as a bias in the study. Extending to a wider spectrum of model families is named as future work.

• Cover missing task domains. Code generation, multi-agent collaboration, and other specialized capabilities are not covered by the current thirteen metrics.

• Extend calibration ablations. The calibration experiments cover only reasoning ability via reasoning-centric datasets, leaving open whether calibration can be designed for safety, multilingual, or instruction-following retention.

• Explain and mitigate the reliability decoupling. The paper establishes that safety, fairness, and privacy do not correlate with performance retention but does not identify why, or what a practitioner should do beyond evaluating those dimensions directly.

Target Audience

Machine learning engineers and practitioners who select and deploy compressed LLMs, particularly those working on on-device, privacy-sensitive, or safety-critical systems. Also relevant to compression researchers who need a broader benchmark than knowledge-centric multiple-choice suites, to evaluation and safety researchers interested in the performance-reliability decoupling, and to students at the intermediate level who already understand pruning, quantization, and distillation at a conceptual level.

Authors’ abstract

Model compression is increasingly essential for deploying large language models (LLMs), yet existing comparative studies largely focus on pruning and quantization evaluated primarily on knowledge-centric benchmarks. Thus, we introduce UniComp, a unified evaluation framework for comparing pruning, quantization, and knowledge distillation. UniComp evaluates compressed models along three dimensions: performance, reliability, and efficiency, using a diverse set of capability- and safety-oriented benchmarks together with a hardware-aware efficiency analysis. Through evaluation of seven compression techniques across over 40 datasets, we observe (i) a consistent knowledge bias, where factual recall is largely preserved while multi-step reasoning, multilingual, and instruction-following capabilities degrade; (ii) a deployment-critical performance-reliability decoupling, where retained performance does not indicate preserved safety, fairness and privacy; and (iii) that task-specific calibration can yield up to 50% relative improvement in reasoning performance in pruned models.