GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compression

AI summary

Overview

• Research area: Post-training (training-free) compression of transformer models via data-aware matrix factorization and cross-layer weight sharing.
• Technical level: Advanced. The method relies on generalized Sylvester equations, generalized eigenvalue decompositions, Cholesky whitening, maximum-weight graph matching, and Hard Thresholding Pursuit.
• Scope: A single paper (arXiv:2609.25963v1 [cs.LG], 22 Sep 2026, CC BY 4.0) by Baher Mohammad, Ammar Ali, and Stamatios Lefkimmiatis (MWS AI, ITMO University; MWS AI) that replaces heuristic cross-layer grouping and covariance merging in transformer compression with an optimization-driven pipeline it calls GeoPair.

What This Paper Is About

Post-training compression pipelines usually factorize each transformer weight matrix on its own, ignoring the redundancies that exist between layers, or they group layers by fixed adjacency and average their activation statistics together. Averaging distorts each layer's own activation geometry, and fixed adjacency misses non-local structural alignments, so fidelity drops at high compression ratios. GeoPair's goal is to pick which layers to pair globally, and to learn a shared dictionary for each pair, while preserving each layer's distinct calibration geometry — all without any fine-tuning or recovery training.

Key Contributions

1. Shared-dictionary learning under distinct whitening spaces. A closed-form generalized Sylvester solver learns a dictionary shared by two layers whose Cholesky whitening transforms differ, removing the heuristic covariance aggregation used in prior cross-layer sharing work.
2. Globally optimal layer pairing. Cross-layer grouping is formulated as a maximum-weight matching problem over a complete graph and solved exactly with Edmonds' Blossom algorithm, using a scale-invariant, shape-agnostic column-space alignment metric rather than a fixed-adjacency rule.
3. Sparse coefficient optimization with convergence guarantees. Hard Thresholding Pursuit (HTP) with a batched conjugate gradient solver enforces an ℓ₀ budget on coefficient matrices, giving adaptive, layer-specific compression without manually allocated budgets or dynamic scheduling; convergence to a block-stationary point is argued under Block Successive Upper-bound Minimization (BSUM) and Kurdyka-Łojasiewicz (KL) theory.
4. Broad empirical validation. Experiments span multiple architectures, parameter scales, and modalities (text and video), comparing against independent factorization, heuristic merging, dictionary-learning, and pruning baselines.

Main Findings

• Ablation on Llama 3 models (CR = 0.2). On Llama3.2 1B, the average accuracy moves from 33.11 for Basis Sharing (Global Whitening) to 41.26 with the Sylvester solver, 43.03 adding Optimal Grouping, and 54.09 with the full pipeline including HTP; the uncompressed baseline is 57.61, so the full configuration recovers over 90% of baseline accuracy. WikiText perplexity falls from 928.07 (Basis Sharing) to 15.92 (full), against a baseline of 11.60.
• Removing optimal grouping hurts. On Llama3.2 1B, the sparsified variant without Optimal Grouping reaches 53.06 average accuracy versus 54.09 with it; on Llama3 8B the same comparison is 67.31 versus 68.17 (baseline 70.36).
• Best training-free method against pruning baselines (Table 2). On Llama2 7B at 20% compression, GeoPair is the only training-free method reaching 63.58 average accuracy, versus 59.82 (LLM-Pruner), 60.05 (LoRAPrune), 57.23 (WANDA), 58.12 (ShortGPT), and 62.22 (LoRAShear); the baseline is 68.59.
• Consistent gains over Basis Sharing across model families (Table 3). For Qwen 3 8B at CR = 0.2 the table lists 67.7 average accuracy versus 61.25 for Basis Sharing; for Gemma 3 12B at CR = 0.3, 67.72 versus 50.2; for Phi-4 14B at CR = 0.2, 74.07 versus 70.33; for Qwen 3 32B at CR = 0.4, 70.31 versus 59.5.
• Small compression can act as denoising. The paper reports minor improvements over the uncompressed baseline at small compression ratios, noting this aligns with recent findings that small rank truncation preserves salient features (e.g., Phi-4 14B average accuracy at CR = 0.2 is 74.07 versus 72.09 at CR = 0).
• Grouping metric ablation (Table 4, Llama3.2 1B, CR = 0.4, KS = 2.0). General Frobenius Norm grouping gives the best trade-off at 34.26 WikiText-2 perplexity and 45.61 average accuracy, versus 43.75/45.0 for cosine similarity and 37.97/43.35 for consecutive-layer greedy grouping (baseline: 11.60 and 57.6).
• KS ratio ablation (Table 5, Llama3.2 1B, CR = 0.4). KS = 2.5 is selected as the default, giving 34.26 WikiText-2 perplexity, 21.69 Lambada perplexity, and 45.6 average accuracy; the sweep covers KS = 2.0, 2.5, 3.0, 3.5, and 4.0 (KS 4.0 yields 35.43, 24.55, and 44.3).
• Video generation preserved. Wan2.2 5B compressed at 20% and 40% with no post-compression fine-tuning retains near-baseline text-video alignment measured with X-CLIP using 16-frame sampling over 50 prompts from the Rapidata/awesome-text2video-prompts dataset: at 20% compression the average CLIP score drops by only 1 × 10⁻⁴ (0.2164 vs. 0.2165 baseline), and at 40% compression it is 0.2111, a 2.5% drop.
• Comparative landscape (Figure 2). On Llama3 8B across compression ratios spanning 0.2 to 0.7, GeoPair reports the highest accuracy and lowest log-perplexity on WikiText compared with Baseline, Basis Sharing, SVDLLM, COSPADI, ROCKET, and COMPOT.

Methodology in Plain English

The starting point is that a weight matrix should be compressed by looking at the activations that flow through it, not just the weights. The authors express compression as minimizing the error in reproducing activations over a small calibration set, and they whiten that problem using a Cholesky factor of a regularized activation Gram matrix (a Tikhonov term, η > 0, guarantees the matrix is positive definite). Because all layers in a group must be compared in a common frame, the same relative scaling is

Authors’ abstract

Transformer architectures exhibit cross-layer redundancies, yet post-training compression pipelines typically optimize layers in isolation or rely on heuristic grouping strategies that disregard layer-specific activation geometries. We introduce a principled, training-free framework that sequentially optimizes cross-layer weight pairings and shared-dictionary factorizations. Rather than forcing weights of adjacent layers to share a basis or heuristically merging activation statistics, our approach identifies structurally compatible projections and learns a shared representation that better preserves each layer's distinct calibration geometry. Coupled with structured sparsity, this yields highly efficient weight decompositions without sacrificing functional fidelity. Across diverse architectures, scales, and modalities, our method achieves state-of-the-art results, consistently outperforming independent structured weight decompositions and alternative pairwise weight factorizations, which operate under heuristic grouping strategies. By replacing heuristic engineering strategies with a convergent, optimization-driven pipeline, we establish a theoretically grounded foundation for scalable, transformer compression across different modalities.