Neural Spectral Capacity: Measuring and Designing Architectures from Network Specification Alone

AI summary

Overview

• Research area: Machine learning — neural architecture design, neural architecture search (NAS), and structured pruning of large language models, approached through random matrix theory and information theory.
• Technical level: Advanced. The paper is grounded in singular-value decomposition, the Marchenko–Pastur law, Shannon transforms, and bounded-knapsack dynamic programming. The prose is readable, but readers without linear algebra and probability background will need to slow down on Sections 3.1 and 3.2.
• Scope: The paper proposes Neural Spectral Capacity (NSC), a closed-form scalar computed from an architecture's specification alone, and the accompanying NSC-DP solver that finds the globally NSC-optimal architecture under a resource budget, evaluated across seven Transformer and CNN families plus LLaMA-7B pruning.

What This Paper Is About

Designing or compressing a Transformer requires deciding how to distribute capacity across depth, width, attention heads, and FFN ratios under a fixed parameter or compute budget. The standard tools for these decisions, #Params and #FLOPs, describe size and compute but not structure: two architectures with identical parameter budgets but different depth-width or head allocations receive identical scores, even though they behave differently. The paper's goal is to define a scalar that captures architectural structure, is computable directly from the network specification (no model instantiation, no data, no gradients), and can be optimized exactly under resource constraints.

Key Contributions

1. NSC, a closed-form architectural scalar. Grounded in the singular-value spectrum of each weight matrix and connected to the mutual information of the corresponding linear Gaussian channel (Telatar, 1999), NSC reduces under standard random initialization to a deterministic expression in matrix dimensions and initialization variance alone via the Marchenko–Pastur law (Theorem 1, Sections 3.1–3.2). Unlike #Params and #FLOPs, it captures depth, head allocation, and FFN ratio.

2. NSC-DP, an exact dynamic-programming solver. Because NSC is layer-wise additive, resource-constrained maximization reduces to a bounded or multiple-choice knapsack solvable exactly by DP (Algorithm 1, Section 3.4). NSC-DP returns the architecture globally maximizing NSC under resource constraints — a guarantee the authors state black-box search (evolutionary, RL, random) over non-decomposable proxies fundamentally cannot provide. The authors also argue only NSC jointly tracks quality and decomposes additively among architectural scalars.

3. Empirical validation across ranking, design, and compression. Seven Transformer and CNN families are used for ranking; Transformer-XL and AutoFormer for end-to-end search; LLaMA-7B for structured pruning. NSC scores architectures in microseconds on CPU.

4. A public code release at https://github.com/Optima-CityU/neural-spectral-capacity.

Main Findings

• NSC discriminates where parameter count collapses. On FlexiBERT, NSC reaches Kendall τ = 0.695 versus 0.485 for #Params and 0.552 for #FLOPs (Table 3). Under the 10%-PW control, which restricts τ to architecture pairs differing in #Params by less than 10%, #Params drops to τ = 0.082 while NSC retains τ = 0.505.

• The 10%-PW advantage generalizes. NSC's 10%-PW τ exceeds #Params' on every benchmark: 0.503 vs. 0.322 on AutoFormer-Tiny, 0.372 vs. 0.116 on NATS-Bench-SSS, and 0.535 vs. 0.414 on MobileNetV3 (Appendix F.2).

• NSC beats training-free proxies that require instantiation. On FlexiBERT, NSC leads W-PCA by +0.060 and ZeroLM by +0.168 in τ; the lead widens on AutoFormer-Tiny (+0.110 over W-PCA), while on GPT-2 NSC is competitive with the leading proxies (within 0.03 of W-PCA) at orders-of-magnitude lower wall-clock. NSC takes 2 ms for all 500 FlexiBERT architectures, versus 88 s (W-PCA), 63 s (ZeroLM), 73 s (SNIP), and 74 s (GradNorm) on a single A100 GPU.

• Spectral capacity is not a restatement of size. A full-rank 4096 × 11,008 matrix from a LLaMA-7B FFN layer has ψ ≈ 3174, while its rank-1 truncation has ψ ≈ 2.5 — a 1,270× drop with m·n unchanged.

• Initialization-time rankings track training. Across 10 GPT-2 architectures on WikiText-103, ψ_t increases monotonically during training, initialization rankings are preserved (ρ(ψ_0, ψ_T) = 1.00), and converged ψ_T ranks final perplexity perfectly (|ρ| = 1.00). The authors note these perfect correlations reflect the small 10-architecture sample with deliberate size spread; larger-scale results report Spearman ρ ∈ [0.78, 0.97]. The derivative ∂ψ/∂σ_i = 2σ_i/(1 + σ_i²) peaks at σ_i = 1, where variance-preserving initialization places the bulk of the spectrum.

• The closed form is tight at practical sizes. The relative error of ψ_MP against the SVD-based value decays as O(1/min(m,n)), falling below 0.4% at min(m,n) = 128 and to ~10⁻⁴ by min(m,n) = 1024. On the LoNAS-LLaMA-7B trained supernet, ψ_MP matches ψ_W at τ = ρ = 1.0000 across all 129 Pareto-optimal subnets, with 0.039% mean SVD-oracle regret and a 720× compute saving over the SVD route.

• Summation is the best aggregate. The aggregation ablation (Table 1, against GLUE, perplexity, and ImageNet Top-1) gives sum rule scores of 0.884 (297 distinct scores) on FlexiBERT, 0.968 (200) on GPT-2, and 0.810 (208) on AutoFormer-T; the min (bottleneck) rule collapses to 0.178 (22), 0.813 (195), and −0.173 (3) respectively.

• NSC-DP finds a better Transformer-XL at trivial cost. On WikiText-103, NSC-DP reaches test PPL 23.087 at 40M parameters in 2.0 seconds on CPU, improving on Transformer-XL Base (23.279, 38M), Synaptic Div. (23.135, 37M, 903 s), Softmax Conf. (23.532, 39M, 883 s), and W-PCA (23.669, 40M, 965 s) — over 400× faster than the training-free proxy baselines.

• NSC-DP matches a day-scale oracle on vision Transformers. On AutoFormer-Tiny (ImageNet-1K), NSC-DP reports Acc@1 75.276 and Acc@5 92.788 at 5.8M parameters in 0.03 s, against the AutoFormer-T oracle (75.308 / 92.690 at 5.7M, 24 GPU-days), TF-TAS (75.234 / 92.730, 0.5 d), AZ-NAS (74.804 / 92.504, 2,592 s), and W-PCA (74.752 / 92.500, 206 s).

• LLaMA-7B pruning is calibration-data-free and far faster. Via the LoNAS SuperNet at a 5.7B budget, NSC-DP prunes LLaMA-7B to the best 5.7B model across eight commonsense reasoning tasks (BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC-Easy, ARC-Challenge, OpenBookQA) with no calibration data, ~5900× faster than the strongest training-free proxy baseline. Baselines (SNIP, GradNorm, SynFlow, W-PCA) each used evolutionary search with population size 50 for 20 generations (1,000 proxy evaluations) on a WikiText-103 calibration batch of B = 2 tokens of length T = 2048; the un-pruned LoNAS SuperNet is 6.7B. The per-task Table 5 numbers are not included in the truncated content provided here, so the individual task accuracies and the Avg₈ values are not reported in this summary.

• Efficiency of the solver. Each ψ_MP evaluation is a 1D numerical quadrature costing ~10 μs, reused from cache when the (m, n, s) tuple repeats; the inner DP runs in O(L(g) · B · |X_l(g)|) on a discretized budget axis. NSC-DP searches spaces of up to 10³² architectures in seconds on one CPU core, and a single run yields the whole Pareto front over the discretized budget grid via O(B) table look-ups.

• Which matrices count. Per-head Q/K/V projections are treated separately rather than as one 3d × d matrix — necessary to discriminate head count — plus the attention output projection and FFN matrices; convolutions are reshaped from c_out × c_in × k × k to c_out × c_in·k². Embeddings, biases, normalization, and nonlinearities are excluded.

Methodology in Plain English

The authors start from a communications analogy. Every linear weight matrix W can be viewed as a channel that transmits an input signal while adding noise, and information theory gives an exact formula for how much information can pass through it: the log-determinant of I + WᵀW, which equals a sum over the matrix's squared singular values. The authors call this quantity the spectral capacity of the matrix.

Computing it directly would require materializing the matrix and running an SVD, defeating the purpose. The key step is that at standard random initialization the squared singular values follow the Marchenko–Pastur distribution, a universal law from random matrix theory that depends only on the matrix's aspect ratio and initialization variance. Integrating the capacity formula against that law produces a closed-form expression in three numbers: the matrix's two dimensions and its initialization variance. Under a fixed initialization scheme such as Xavier or Kaiming, the variance is itself determined by shape, so the score becomes a function of the architecture specification alone.

Matrix scores are then summed within a layer (the parallel composition rule for channel capacities, which is what makes per-head decomposition meaningful) and summed across layers to give the network score NSC. The authors show empirically that summation is the best across-layer aggregation, ahead of bottleneck and product rules.

Because NSC is additive across layers, and because typical deployment constraints (parameters, FLOPs) are sums of per-layer costs, choosing the best architecture under a budget becomes a knapsack problem: fix the network-level choices (depth, embedding size), then solve exactly for the best per-layer allocations by dynamic programming, looping over all network-level configurations. This yields NSC-DP, which is exact with respect to the NSC objective rather than merely returning the best candidate found by search.

Evaluation has three parts: rank correlation of NSC against trained performance on architecture pools from seven families; end-to-end search on Transformer-XL and AutoFormer compared to the original baselines and to training-free proxies used as evolutionary fitness; and structured pruning of LLaMA-7B through the LoNAS SuperNet with a zero-shot commonsense evaluation protocol following Munoz et al. (2024).

Why This Matters

The paper targets a gap in the standard toolkit for architecture decisions. #Params and #FLOPs are identical across the very candidates a practitioner is choosing between at a fixed budget, so they cannot inform the choice. Existing training-free proxies can, but they require instantiating a randomly initialized network and measuring something on sampled inputs, their scores depend on which inputs are used, and optimizing them requires black-box search that returns the best architecture visited rather than the best possible under the proxy. NSC is presented as satisfying all of these at once: spec-only, closed-form, microsecond-scale on CPU, and exactly optimizable.

Real-world applications:

• Foundation-model pretraining budget allocation: choosing depth-width tradeoffs, FFN ratios, head sharing (GQA/MQA), and per-layer expert allocation in MoEs before committing to a costly training run.
• Structured pruning of frontier LLMs for deployment: selecting which blocks, FFN widths, and LoRA ranks to retain under a memory or latency target, without access to calibration data — relevant when pretrained weights or representative data are unavailable or restricted.
• On-device and edge model design: quickly generating Pareto fronts across a discretized budget grid from a single solver run, rather than re-running a search per deployment target.
• NAS in compute-constrained research settings: searching spaces of up to 10³² architectures on a single CPU core with no GPU, which lowers the hardware barrier to architecture search work.

Industry relevance: The two costliest decisions in production LLM work are pretraining a new configuration and compressing an existing checkpoint for serving. The paper's claims — that the LLaMA-7B pruned model is the best 5.7B model across eight commonsense tasks without calibration data, and that the whole search takes seconds on CPU rather than proxy search on GPUs — speak directly to both.

Future Directions

• Extending beyond linear-projection matrices. NSC currently excludes biases, normalization, nonlinearities, and embeddings. Whether scalable, exact scores can be defined for these components — and whether doing so improves ranking — is left open.
• Grounding the cross-layer summation rule. The authors explicitly state that summing across layers is an empirically motivated modelling choice, not a claim about the end-to-end mutual information of the forward pass, and support it only by the Table 1 ablation. A principled composition rule for sequentially executed layers is a natural next step.
• Validating at larger scale and on more model families. The near-perfect correlations in Figure 1 come from only 10 architectures with deliberate size spread; the broader results cite Spearman ρ ∈ [0.78, 0.97]. Whether the ranking advantage holds for larger pools, MoE architectures, and other modalities is not yet established.
• Sensitivity to the initialization protocol. Comparing architectures via the closed form requires one fixed initialization scheme across all candidates, since s is otherwise determined by shape and role. The paper reports that the choice of fixed scheme broadly preserves rankings (Appendix H), but the limits of that robustness across schemes and unusual shapes remain a question.

Target Audience

This paper is most useful to machine learning researchers and engineers working on neural architecture search, Transformer design, or LLM compression — particularly those who need a quality-tracking signal that works before any weights, data, or gradients are available, and who want an exactly optimized objective rather than a heuristic search. It also speaks to readers interested in applications of random matrix theory and information theory to deep learning, and to practitioners making depth-width, head-allocation, or pruning decisions under fixed parameter or compute budgets. Readers without linear algebra and probability background will find Sections 3.1–3.2 demanding, though the experimental sections are self-contained.

Authors’ abstract

Modern Transformer design and compression both reduce to allocating capacity under a budget. The standard scalars for these decisions, #Params and #FLOPs, capture size and compute but not architectural structure: two architectures with identical parameter budgets but different depth-width, head, or FFN allocations receive identical scores yet behave differently. We propose Neural Spectral Capacity (NSC), a closed-form scalar grounded in the singular-value spectrum of each weight matrix. Under standard random initialization, the Marchenko-Pastur law renders NSC computable from the architectural specification alone, with no model instantiation, data, or gradients. Its layer-wise additive structure admits NSC-DP, an exact dynamic-programming solver returning the architecture globally maximizing NSC under resource constraints in seconds on a CPU -- a guarantee that black-box search over existing training-free proxies cannot provide. Empirically, NSC outperforms #Params, #FLOPs, and representative training-free proxies in ranking across seven Transformer and CNN families (on FlexiBERT, $τ= 0.505$ on pairs differing in #Params by less than 10%, where #Params collapses to 0.082); NSC-DP discovers a Transformer-XL architecture on WikiText-103 that beats the human-designed baseline in 2 seconds; and prunes LLaMA-7B to the best 5.7B model across eight commonsense reasoning tasks without any calibration data, about 5900x faster than the strongest training-free proxy baseline.