Beyond Adjacent Layers: Graph-Guided Layer Fusion for Compressing Large Language Models

AI summary

Overview

Research area: Machine learning — specifically model compression for large language models (depth compression / layer pruning and merging).

Technical level: Advanced. The paper assumes familiarity with transformer block structure, Centered Kernel Alignment (CKA), spectral graph clustering, LoRA fine-tuning, and standard perplexity benchmarking.

Scope: A 13-model empirical study showing that functionally redundant layers in LLMs are frequently non-adjacent, plus a new compression method (Fusion via Community Mining, FCM) that exploits this global similarity structure via graph community detection.

What This Paper Is About

Almost all existing methods for compressing an LLM by deleting layers assume that redundant layers sit next to each other: they either score layers one at a time (keeping the individually "most important" ones) or merge only adjacent/contiguous blocks. The authors test that assumption directly by computing full layer-by-layer similarity matrices for thirteen LLMs, and find that highly similar layer pairs are far more often non-adjacent than adjacent. They then build a compression method that treats the layers as a graph, finds clusters ("communities") of functionally equivalent layers that may be spread across the whole depth, and keeps one high-impact layer from each cluster.

Key Contributions

1. An empirical characterization of non-local redundancy. Full pairwise CKA similarity matrices for thirteen decoder-only LLMs from six architecture families (1.6B–14B parameters) show that non-adjacent high-similarity layer pairs outnumber adjacent ones by 3.6× to 15.9×, a structure that adjacency-restricted and contiguous-segmentation methods cannot represent.

2. The FCM method. Fusion via Community Mining builds the full L×L similarity graph, applies spectral community detection to partition layers into K communities (K fixed by the target compression ratio), retains the highest Block Influence (BI) layer per community, and heals the result with LoRA recovery. It requires no similarity cutoff to form groups and no training beyond LoRA.

3. A calibration-only metric-selection rule. A pre-specified "partition degeneracy" test, computed purely from the calibration similarity matrix (size-balance entropy below β*=0.70 or maximum coverage above μ*=0.40), decides whether to use cosine or CKA similarity — explicitly avoiding test-set tuning. The thresholds sit in a wide invariance band (μ* ∈ [0.225, 0.455], β* ∈ [0.635, 0.870] give identical routing across all thirteen models).

4. A broad controlled evaluation. Head-to-head comparisons at 2× compression against ShortGPT, contiguous block removal, FlattenGPT, SWM, DeltaLLM, and MKA under matched calibration and matched LoRA recovery, plus a compression sweep from 1.5× to 3.0×, downstream benchmarks, and inference profiling.

Main Findings

• Non-adjacent similarity dominates. In Qwen2.5-7B, 125 non-adjacent layer pairs exceed CKA 0.8 versus only 26 adjacent pairs (4.8×). The ratio spans 15.9× (SmolLM3-3B) down to 3.6× (Qwen3-4B), and holds at 14B scale (Qwen2.5-14B: 4.35×; Qwen3-14B: 4.27×).

• "Bookend" communities exist. CKA matrices show block-diagonal structure where early embedding layers and late output layers form a single functional community despite being separated by the entire middle block — for Qwen2.5-7B, layers {0,1,2} pair with {25,26,27}.

• Beats ShortGPT on 12 of 13 models at 2× compression. Ten of these are significant under a Welch's t-test with Holm–Bonferroni correction, with reductions up to 50.0% (Qwen3-4B, p_corr < 0.001, Hedges' g = 5.92) and 40.7% (Qwen2.5-7B, g = 4.85). SmolLM3-3B (−4.7%, p_corr = 0.084) and LLaMA-3.2-3B (−5.0%, p_corr = 0.154) are ties; Mistral-7B is the only loss (+13.4%).

• Best against adjacent-pair methods. Against FlattenGPT, SWM, and DeltaLLM under matched calibration and recovery, FCM has the best perplexity on 10 of 13 models (mean rank 1.15 vs. 2.15 for DeltaLLM, 3.38 FlattenGPT, 4.08 SWM, 4.23 ShortGPT), with margins over the strongest baseline of 7.8% to 31.2%. DeltaLLM is the strongest baseline (best non-FCM method on 11 of 13). FCM also beats the similarity-based merger MKA on three anchor models.

• Gains transfer downstream. On a five-benchmark zero-shot suite (ARC-Easy, ARC-Challenge, HellaSwag, WinoGrande, PIQA), FCM matches or exceeds ShortGPT on 15 of 20 (model, benchmark) cells, and beats it on every cell for the Qwen and LLaMA anchors (15 of 15), recovering up to 6.2 points of mean accuracy on Qwen2.5-7B. It also improves MMLU (5-shot) by 6.2–10.6 points over ShortGPT, halves LAMBADA generative perplexity (8.65 vs. 18.42 on Qwen2.5-7B), raises HumanEval pass@1 to 29.8% vs. 18.5%, and lifts MAUVE from 0.615 to 0.894 while cutting repetition from 24% to 2%.

• The advantage grows with compression. A crossover appears at 2×: below it (1.5×, where all methods retain 65–67% of layers) DeltaLLM or ShortGPT edge FCM by ≤ 0.6 PPL; beyond it FCM's lead widens monotonically. On Qwen2.5-7B the FCM-vs-DeltaLLM gap grows from −8.95 PPL at 2× to −125.9 PPL at 3×, where ShortGPT exceeds 380 PPL, and FCM's zero-shot gain over ShortGPT stays at +6.2 to +9.7 points.

• Real efficiency wins. The 2×-compressed Qwen2.5-7B has 46% fewer parameters, 42% lower peak VRAM, and 84% higher decode throughput on a single A100.

• Grouping and selection choices both matter. Ablation on Qwen2.5-7B shows spectral grouping beats Fisher contiguous segmentation (48.92 vs. 82.93) and uniform partitioning (48.92 vs. 99.63) — reported as 41% and 51% improvements. Cosine beats CKA for grouping (35.01 vs. 48.92). Keeping one intact layer beats weight averaging (35.01 vs. 57.23). BI-guided selection beats centrality selection (26.65 vs. 35.01 on Qwen2.5-7B; 35.25 vs. 449.88 on LLaMA-3.1-8B).

• Contiguity itself is costly. A contiguous-block-removal baseline on Qwen2.5-7B reaches 71.02 PPL, worse than ShortGPT's 44.92 and FCM's 26.65.

• Overhead is small. Profiling costs about 11.5 seconds on Qwen2.5-7B (L=28, N=512, d=3584) with clustering under 0.01 seconds, versus roughly 840 seconds for 500 LoRA steps — about 1.4% added to the offline pipeline versus ShortGPT.

Methodology in Plain English

The pipeline has three stages.

1. Measure how similar each pair of layers is. The authors feed 512 calibration samples from C4 (sequence length 128) through the uncompressed model and record the average hidden state each layer produces. They then compute a similarity score for every pair of layers — not just neighbours — producing a full L×L table. Two metrics are considered: CKA (an aggregate, kernel-based alignment score) and mean cosine similarity (a per-sample score). By default the method uses cosine, switching to CKA only when a calibration-only degeneracy check flags the cosine partition as badly unbalanced.

2. Find communities of similar layers. The similarity table is treated as a weighted graph: layers are nodes, similarity scores are edge weights. Spectral clustering splits the graph into K groups, where K is set directly by the target compression ratio (halve the layers → K = L/2). Because the graph ignores layer position, a community can contain the first, third, and last layers if they behave alike.

3. Keep one representative per community. Within each community, the layer with the highest Block Influence — the cosine distance between its input and output, i.e. how much it actually changes its input — is kept and the rest are discarded. Retained layers keep their original weights untouched, and the compressed model is then healed with LoRA adapters on all linear projections (rank 16, alpha 32, dropout 0.05, 500 steps, learning rate 2×10⁻⁴, AdamW without weight decay). Every baseline receives identical recovery, isolating the grouping decision.

Evaluation is WikiText-2 perplexity at primarily 2× compression, with seeds 42/123/7 for three-seed variance and additional seeds 2024/999 for nine models. Statistics use a two-sided Welch's t-test with Holm–Bonferroni correction. Experiments up to 8B run on a single NVIDIA RTX PRO 6000 (96 GB); 14B runs use NVIDIA A100 80GB PCIe GPUs. Code is at https://github.com/Usama1002/fcm-layer-fusion.

The intuition for why this works: if you just drop the lowest-scoring layers, you may accidentally delete everything from one functional phase (say the middle) and keep nothing representative of the early or late phases. Picking one layer per community forces coverage of every phase.

Why This Matters

For research: The paper directly challenges an implicit assumption baked into a whole family of depth-compression methods (ShortGPT, Gromov et al., LLM-Streamline, FlattenGPT, SWM, DeltaLLM, SGLP) — that redundancy is local. The 3.6–15.9× ratio replicates across six architecture families, four scale tiers, and multiple training pipelines, which suggests it is a geometric property of the Pre-LayerNorm residual stream rather than a quirk of one model. The finding that centrality-based selection catastrophically fails on LLaMA-3.1-8B (+948% over ShortGPT) while BI-guided selection on the same communities wins (−17.5%) is a useful methodological lesson: the most redundant layer in a cluster is the worst one to keep.

Real-world applications:

• Serving LLMs under tight GPU memory budgets, where a 42% peak-VRAM reduction on a single A100 changes what fits on a node.
• Latency-sensitive products (chat, code completion, agent loops) where 84% higher decode throughput on the compressed Qwen2.5-7B translates into more concurrent users per accelerator.
• Edge and on-device deployment, where halving depth roughly halves sequential dependency chains.
• Composing with orthogonal compression families named in the paper — width pruning (SliceGPT, SparseGPT, Wanda) and post-training quantization (GPTQ, AWQ) — to stack savings.

Industry relevance: The method needs no retraining from scratch, only LoRA recovery, and adds roughly 1.4% to the offline pipeline cost over a ShortGPT baseline. That makes it a drop-in candidate for teams already using layer-pruning recipes. The explicit design against test-set tuning (a calibration-only routing rule, evaluated over invariance bands) is also relevant to practitioners who need defensible, reproducible compression decisions.

Future Directions

• Diagnosing and fixing the Mistral-7B failure. Mistral-7B is the one loss (+13.4% mean-pooled; +5.3% with full-token CKA profiling), and the paper attributes this to sliding-window attention. A principled treatment of position-conditional similarity for sliding-window architectures is left open.
• Making community detection avoid "bad" communities. The LLaMA-3.1-8B result shows grouping is only as good as the community structure; methods for detecting and splitting communities dominated by redundant layers, beyond swapping in a BI-guided selector, are not developed here.
• Extending past 3× compression and past 14B. The advantage grows monotonically up to 3×, but behaviour at 4× and above, and at scales beyond 14B, is not reported.
• Composition with width pruning and quantization. The paper positions FCM as orthogonal to SliceGPT/SparseGPT/Wanda and GPTQ/AWQ, but no combined experiments are reported.

Target Audience

Researchers and engineers working on LLM inference efficiency and model compression, particularly those already familiar with layer pruning, LoRA recovery, and perplexity-based evaluation. It is also valuable for anyone studying representational similarity in transformers (the CKA analysis stands on its own), and for ML systems practitioners who need a compression method that composes with quantization and width pruning rather than replacing them. Readers without a background in spectral clustering or CKA will need to consult the cited references (Ng et al., 2001; Kornblith et al., 2019) to follow the methodology in detail.

Authors’ abstract

Existing depth-compression methods for large language models (LLMs) score layers individually or group only adjacent or contiguous layers, implicitly assuming that functional redundancy is local. We test this assumption by computing full pairwise Centered Kernel Alignment (CKA) similarity matrices for thirteen LLMs from six architecture families (1.6B-14B parameters). Non-adjacent layers have 3.6-15.9x more high-similarity pairs than adjacent ones, a structure that prior similarity-based layer merging has not systematically exploited. We propose Fusion via Community Mining (FCM), which builds the full L x L similarity graph, uses spectral community detection to find clusters of functionally equivalent layers, keeps the highest-Block-Influence layer in each cluster, and applies LoRA recovery; a pre-specified rule computed only on calibration data chooses between cosine and CKA similarity. At 2x compression, FCM lowers mean perplexity below ShortGPT on twelve of thirteen models, significantly on ten (Welch's t-test with Holm-Bonferroni correction; up to 50% on Qwen3-4B), with Mistral-7B the only loss (+13.4%). Against FlattenGPT, SWM, and DeltaLLM under matched calibration and recovery, FCM gives the best perplexity on ten of thirteen models (mean rank 1.15), and it also outperforms MKA on three anchor models. FCM matches or exceeds ShortGPT on 15 of 20 zero-shot reasoning cells and also improves MMLU, LAMBADA, HumanEval pass@1, and free-form generation quality. The 2x-compressed Qwen2.5-7B has 46% fewer parameters, 42% lower peak VRAM, and 84% higher decode throughput on a single A100. Spectral grouping outperforms Fisher contiguous segmentation by 41% and uniform partitioning by 51%, and FCM's advantage grows beyond a 2x crossover up to 3x and persists at the 14B scale. Code: https://github.com/Usama1002/fcm-layer-fusion