SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference

AI summary

Overview

Research area: Efficient large language model inference — specifically training-free post-training weight pruning (Hessian/OBS-based methods) combined with custom sparse GPU kernel design for autoregressive decoding.

Technical level: Advanced. The paper assumes familiarity with the Optimal Brain Surgeon (OBS) pruning framework, Hessian-based layer-wise reconstruction, N:M semi-structured sparsity, and GPU sparse matrix kernels (SpMM vs. SpMV).

Scope: The paper proposes SparseDecoding, a framework that builds pruning calibration matrices from a dense model's own autoregressive decode-step activations (discarding prefill) and pairs it with a purpose-built N:M SpMV kernel, evaluated on four open-weight LLMs across writing and code-generation benchmarks.

What This Paper Is About

LLM decoding is memory-bound and slow, and pruning is a common fix because it reduces the number of nonzero weights read from memory. However, standard Hessian-based pruning calibrates on pre-collected natural text, while decoding actually feeds the model its own self-generated tokens — a distribution shift. The paper's motivating measurement is that the activation discrepancy between dense and pruned models grows rapidly during early decoding, with all examined layers reaching at least 90% of their eventual plateau within the first 12 decoding steps, so the Hessian computed on natural sequences does not match the one computed on generated sequences.

Separately, existing sparse libraries that deliver real speedup target sparse matrix-matrix multiplication (SpMM), which suits prefill, while decoding is dominated by sparse matrix-vector multiplication (SpMV). The paper's Figure 1(b) reports that on Llama-3.1-8B with 2:4 sparsity on an NVIDIA A100 GPU, existing 2:4 Sparse Tensor Core kernels achieve 1.31 to 1.46× speedups during prefill but only 0.85 to 0.87× of dense performance during autoregressive decoding. The goal is to fix both the calibration mismatch and the kernel mismatch.

Key Contributions

1. A decoding-aware pruning method that aligns the pruning objective with the activations encountered during autoregressive decoding, by constructing calibration matrices from layer-wise activations collected while the dense model generates tokens, keeping only decode steps and discarding prefill activations. The method works directly with existing OBS-based solvers such as SparseGPT and requires no additional training or a new optimizer.

2. An optimized N:M SpMV kernel for efficient sparse decoding, using bitmask indexing and fixed-step traversal to reduce metadata and indexing overhead, inspired by the nmSPARSE library.

3. A theoretical characterization of the discrepancy between a calibration objective and the decode-time reconstruction objective. The paper states that Theorem A.1 measures the exact worst-case relative discrepancy, and Definition A.1 defines a normalized decode-time calibration discrepancy that is zero when the calibration Hessian exactly matches the decode-time Hessian after projection onto the top-k decode-time eigenvectors and eigenvalue normalization.

4. Extensive experiments across multiple LLMs showing improved long-form generation quality over existing pruning baselines, together with up to 1.48× end-to-end decoding speedup on A100 GPUs.

Main Findings

• Calibration mismatch is real and early: The relative activation discrepancy between dense and pruned models grows rapidly during the first few decoding steps, with all layers reaching at least 90% of their plateau by decoding step 12, after which it stays largely unchanged even over hundreds of additional steps. The figure shows the first (layer 0), middle (layer 16), and last (layer 31) layers, with shading for the interquartile range across transformer layers.

• Decoding-aware calibration improves long-form writing quality: On WritingBench, SparseDecoding improves the overall score across all four models and both sparsity settings. Under 50% unstructured sparsity, gains range from 0.19 to 0.75 points. Under 2:4 sparsity, the overall score rises from 1.34 to 1.52 for Llama-3.1-8B, from 3.10 to 3.29 for Llama-3.3-70B, from 1.85 to 4.18 for Qwen3-14B, and from 2.99 to 5.31 for Qwen3-32B.

• Largest gains appear on the Qwen models under 2:4 sparsity: While Llama gains under 2:4 are described as modest, the Qwen models improve substantially — for example, Qwen3-32B moves from 2.99 (C4) to 5.31 (SparseDecoding). Dense references on WritingBench are 3.70 (Llama-3.1-8B), 4.65 (Llama-3.3-70B), 5.97 (Qwen3-14B), and 6.48 (Qwen3-32B).

• Code generation improves on all eight model-sparsity pairs: On ClassEval class-level Pass@1, SparseDecoding is ahead of C4 by 4 to 15 points across all eight pairs. Under 2:4 sparsity, Pass@1 rises from 9.0 to 24.0% on Qwen3-32B, from 0.0 to 6.0% on Qwen3-14B, from 11.0 to 15.0% on Llama-3.3-70B, and from 0.0 to 4.0% on Llama-3.1-8B — meaning SparseDecoding recovers nonzero scores where C4 yields 0.0.

• Every evaluated N:M pattern decodes faster than dense: Across 2:4, 4:8, 8:16, and 16:32 (all retaining 50% of weights), speedups range from 1.34 to 1.48×. Headline per-model speedups are 1.42× on Llama-3.1-8B, 1.48× on Llama-3.3-70B, 1.35× on Qwen3-14B, and 1.45× on Qwen3-32B. Dense throughput baselines are 96.26, 21.67, 39.76, and 19.15 tokens/s respectively; the best sparse throughputs are 136.89, 32.13, 53.86, and 27.83 tokens/s.

• Near-identical throughput across patterns is expected: Because all patterns retain 50% of weights and use the same bitmask metadata, they incur identical memory traffic per row. This contrasts with the 2:4 Sparse Tensor Core path, which runs slower than dense during decoding.

• Metadata reduction: At 50% sparsity, the bitmask representation uses only 2 bits of metadata per nonzero, versus 32 bits for the int32 indices used by nmSPARSE and 8 bits for a uint8 offset representation — a reduction of 16× and 4× respectively.

• Self-generated calibration beats cross-model calibration: Pruning Qwen3-14B with activations from its own generations scores 5.35 versus 5.23 when using Qwen3-32B generations under 50% sparsity, and 4.18 versus 4.13 under 2:4 sparsity, suggesting cross-model calibration introduces a distributional discrepancy.

• Gains are not specific to the pruning backend: With Wanda instead of SparseGPT, SparseDecoding remains ahead of C4 in both sparsity settings on both Qwen3 models (for example, Qwen3-14B: 5.23 vs 4.94 at 50%, and 3.30 vs 2.41 at 2:4; Qwen3-32B: 5.71 vs 5.52 at 50%, and 4.39 vs 3.34 at 2:4).

Methodology in Plain English

Step 1 — Change what the calibration data are. Standard one-shot pruning picks a small text corpus such as C4, runs the model over it in a teacher-forced way, and records the input activations to each linear layer. Those activations define a Hessian, which decides which weights to remove and how to adjust the survivors. SparseDecoding instead takes a set of prompts from the target task, lets the dense model generate its own continuations token by token, and records layer input activations only at the decoding steps. The activations from the prefill pass over the prompt are thrown away. The resulting matrix of decode-step activations becomes the calibration matrix in the same layer-wise reconstruction loss, and it is handed to the same unmodified pruning solver.

Step 2 — Replace index metadata with bitmasks. In the sparse kernel, rather than storing a separate 32-bit column index for every retained weight, the pattern of each row is stored as a bitmask with one bit per input column, packing every 32 consecutive bits into a 32-bit mask word. Because weights are stored in increasing column order, the set bits in the mask correspond one-to-one to the stored weights, so no separate column index or local offset is needed.

Step 3 — Make the traversal uniform. For a general sparse matrix, rows have different numbers of nonzeros and therefore take different numbers of traversal steps. Under 50% N:M sparsity with M ≤ 32, each aligned 32-bit mask word contains exactly 16 set bits, so every row requires the same number of steps, known at compile time. Each step uses a bit-scan operation to find the least significant set bit (giving the column of the next nonzero) and then clears that bit. In the paper's framing, this fixed-length loop keeps the workload uniform across GPU threads.

Step 4 — Evaluate. The paper prunes four instruction-tuned open-weight models (Llama-3.1-8B, Llama-3.3-70B, Qwen3-14B, Qwen3-32B) once after calibration, using 2:4, 8:16, and 50% unstructured sparsity (all retaining 50% of weights). C4 calibration and SparseDecoding calibration both use 1M calibration tokens, with only decode-step tokens counted for SparseDecoding. Prompt sources are LongWriter for writing and LiveCodeBench problem statements (without solutions or reasoning traces) for code. All models operate in non-thinking mode during calibration and evaluation, and both configurations pass activations to the same SparseGPT backend. Quality is measured with WritingBench (judged by DeepSeek-V4-Flash on a 1–10 scale) and ClassEval (class-level Pass@1), and efficiency is measured with GPT-Fast at a context length of 512, reporting the median after 50 warmup and 50 measured iterations.

Why This Matters

Impact on research. The paper reframes calibration data for LLM pruning as a question not just of which corpus to use — the focus of prior work comparing C4 against other sources — but of which inference regime the calibration activations come from. It provides a theoretical quantity (the normalized decode-time calibration discrepancy) for measuring how far a calibration objective sits from the true decode-time reconstruction objective, and it shows that the pruning solver itself does not need to change. It also pairs this with a systems contribution, since the paper argues that reducing theoretical compute does not automatically reduce latency.

Real-world applications:

• Long-form document drafting, where generations can run to 16,000 tokens and decoding latency dominates the user experience.
• Code generation assistants, evaluated here through Python class generation tasks, where degraded quality after aggressive sparsification is especially costly.
• On-device or cost-constrained serving of open-weight models, where weight sparsity reduces memory reads per token and therefore per-token latency.
• Model deployment pipelines that already use training-free, one-shot post-training pruning and could substitute activation collection without changing their toolchain.

Industry relevance. The two contributing affiliations (Westlake University and Ant Group) point at a deployment-oriented problem: serving large open-weight models cheaply. Because SparseDecoding is training-free, requires no new optimizer, and drops into an existing SparseGPT-style backend, the algorithmic change is close to a data-collection change. The kernel work targets the specific regime — small-batch SpMV — where commodity sparse libraries currently underperform dense execution (0.85 to 0.87× on 2:4 per Figure 1(b)).

Future Directions

• Extending the fixed-step kernel beyond the current scope. The paper's fixed-step traversal argument is stated for the 50% N:M patterns with M ≤ 32 considered in the kernel design, where each aligned 32-bit mask word has exactly 16 set bits. How the design generalizes to other sparsity levels, to M > 32, or to less regular patterns is not reported.

• Testing whether calibration prompt volume and task match matter. The paper fixes calibration at 1M tokens and uses LongWriter prompts for writing and LiveCodeBench problem statements for code, but does not report a sensitivity study over calibration token count or over mismatched prompt sources.

• Understanding why the two model families diverge so sharply under 2:4 sparsity. Llama gains are described as modest (for example, 1.34 to 1.52 on Llama-3.1-8B) while Qwen3-14B moves from 1.85 to 4.18. The paper reports the gap but the truncated content does not explain its cause.

• Applying the discrepancy measure more broadly. Definition A.1 and Theorem A.1 give a per-layer worst-case relative discrepancy over the top-k decode-time eigenvectors. Whether this quantity can be used to select calibration sources, layers, or sparsity levels automatically is not reported.

• Broadening beyond the two evaluated benchmarks and four models. The experiments cover WritingBench and ClassEval on Llama-3.1-8B, Llama-3.3-70B, Qwen3-14B, and Qwen3-32B; other tasks, modalities, and model families are not reported.

Target Audience

Researchers and engineers working on LLM inference efficiency, particularly those already familiar with one-shot or post-training pruning (SparseGPT, Wanda, OBS-style solvers) and with GPU sparse kernel implementation. It is most useful to practitioners who need wall-clock decoding speedups rather than only reduced parameter counts, and to researchers interested in how calibration data distribution affects pruned model quality. Readers seeking a beginner-level introduction to pruning or to GPU kernel design would find the paper's assumptions and notation demanding, since it moves directly between Hessian-based reconstruction and low-level SpMV implementation details.

Authors’ abstract

The memory-bound nature of the decoding stage of large language model (LLM) inference incurs significant latency. Layer-wise training-free network pruning approaches guided by the Hessian have been a prominent solution to this problem, as pruning reduces the number of nonzero parameters read from memory during decoding. Nevertheless, typical methods in this line compute the Hessian using pre-collected natural sequences, whereas the model is fed self-generated tokens during decoding, creating a distribution shift between the two sequences. The Hessian calculated on the natural sequence is different from that calculated on the generated sequence. We observe that this discrepancy causes the activation distribution during generation to deviate from that used for pruning, further hurting the pruned model performance. Moreover, most existing LLM pruning methods that bring actual speedup primarily target the sparse matrix-matrix (SpMM) multiplication, providing limited support for the sparse matrix-vector (SpMV) operations, which dominate decoding. To solve these problems, we introduce SparseDecoding, a principled decoding-aware pruning framework tailored for accurate and efficient LLM decoding. Specifically, at the algorithmic axis, SparseDecoding constructs calibration matrices from layer-wise activations collected during the dense-model autoregressive generation, excluding prefill, thereby aligning the pruning objective with the decoding activations. At the system axis, we develop an optimized N:M sparse matrix-vector kernel with bitmask indexing and fixed-step traversal. Substantial empirical results on representative LLMs (Llama-3.1-8B, Llama-3.3-70B, Qwen3-14B / 32B) demonstrate that our method consistently outperforms standard fixed-text calibration on the long-form generation benchmarks while achieving up to 1.48x end-to-end wall-clock decoding speedup on A100 GPUs.