Spectral Geometry for Deep Learning: Compression and Hallucination Detection via Random Matrix Theory

AI summary

Overview

Research area: Machine learning — specifically the use of spectral geometry and Random Matrix Theory (RMT) to analyze the internal activations of deep neural networks, with applications to reliability (hallucination and out-of-distribution detection) and efficiency (model compression).

Technical level: Intermediate overall, with advanced mathematical foundations. The applied framing is accessible, but the background chapters require comfort with linear algebra, eigenvalue decompositions, and probability theory (Marčenko–Pastur, Wigner semicircle, Tracy–Widom, Baik–Ben Arous–Péché transitions).

Scope (one sentence): This is a master's thesis that proposes a single spectral-geometry framework for studying hidden-layer activations, and derives two practical methods from it — EigenTrack for real-time hallucination/OOD detection and RMT-KD for random-matrix-guided knowledge distillation and compression.

Note on completeness: the paper content provided is truncated partway through Chapter 3 (Related Work). Chapters 4 and 5 are described in the thesis summary and introduction, but their experimental setups, benchmark names, dataset sizes, and numeric results are not included in the supplied text. Where that is the case, this summary says so rather than supplying figures.

What This Paper Is About

Modern large language models and vision-language models face two persistent problems: they are unreliable (they hallucinate, misbehave on out-of-distribution inputs, and are poorly calibrated) and they are inefficient (billions of parameters make them expensive to train, serve, and deploy). This thesis argues that both problems can be attacked with the same mathematical tool — the eigenvalue spectrum of hidden activations — because that spectrum reveals whether a model is encoding structured, causal information or drifting into noise-like behavior. The goal is to turn those spectral statistics into two concrete systems: a detector that flags failures in real time, and a compression method that keeps only the statistically meaningful directions in a network.

Key Contributions

1. A unifying spectral framework. The thesis treats eigenvalue statistics of hidden activations as a compact, interpretable description of representation quality, and links reliability and efficiency through the same spectral lens: redundancy is measured by spectral entropy, degeneracy, and deviation from random-matrix baselines, while stability is measured by spectral gaps and outlier eigenvalues.

2. EigenTrack — real-time hallucination and OOD detection. A detector for large language and vision-language models that converts streaming hidden activations into spectral signatures — entropy, eigenvalue gaps, and divergence from the Marčenko–Pastur baseline — and models their temporal evolution with lightweight recurrent classifiers, enabling early detection of reliability failures before they surface in model outputs.

3. RMT-KD — random-matrix knowledge distillation for compression. A compression method that identifies outlier eigenvalues in activation spectra as carriers of causal information, then progressively projects the network onto lower-dimensional subspaces while preserving accuracy through iterative self-distillation. The resulting models are described as significantly more compact and energy-efficient, yet remain dense and hardware-friendly.

4. A grounded survey and synthesis. The thesis organizes prior work on hallucination detection into black-box, grey-box, white-box, and spectral families, and compares traditional compression approaches (pruning, quantization, sparsity, low-rank approximation, and standard knowledge distillation) against statistically grounded criteria, positioning RMT-based selection as an alternative to ad hoc thresholds.

Main Findings

• Spectral statistics separate signal from noise. The central empirical claim of the thesis is that eigenvalue statistics of hidden activations can distinguish structured, causal representations from noise-like variability, providing a stable and generalizable monitoring signal.

• The Marčenko–Pastur bulk is a usable null model. Activations that are approximately mean-zero and isotropic at a given scale produce covariance spectra consistent with random-matrix predictions; directions near the bulk edge are more likely to be correlational or redundant, while eigenvalues that detach from the bulk (outliers) track semantic or task-relevant factors.

• BBP theory supplies a principled threshold. The Baik–Ben Arous–Péché transition states that a spike only emerges from the Marčenko–Pastur bulk when its strength exceeds a critical threshold (equivalently, when the sample eigenvalue exceeds the upper edge λ₊ = σ²(1 + √c)²); below that threshold the associated sample eigenvalue stays buried in the bulk. This is what lets RMT-KD replace heuristic cutoffs with a statistically derived criterion.

• Early detection is possible. Because EigenTrack models the temporal evolution of spectral features rather than final outputs, the thesis reports that reliability failures can be detected before they manifest in generated text.

• Reported performance is qualitative in the available text. The thesis summary states that EigenTrack "achieves state-of-the-art performance across diverse architectures" and that RMT-KD "attains state-of-the-art trade-offs between compression and accuracy," outperforming traditional pruning and heuristic low-rank methods. No benchmark names, dataset sizes, model sizes, accuracy figures, compression ratios, or latency numbers appear in the provided content; those results are reported in Chapters 4 and 5, which are not included here.

• Compression stays hardware-friendly. Unlike pruning, which produces irregular sparsity, the RMT-KD projection yields models that remain dense, which the thesis presents as an advantage for practical deployment.

Methodology in Plain English

The thesis works from a simple observation: when you feed data through a neural network, the hidden activations across many samples can be stacked into a matrix, and the covariance of that matrix has a spectrum. If the activations were pure noise, that spectrum would follow known random-matrix laws — the Marčenko–Pastur distribution for covariance-type matrices, and the Wigner semicircle for symmetric random matrices. Anything that sticks out beyond that baseline — a detached outlier eigenvalue, a widened gap, unusual entropy — is informative structure.

For detection (EigenTrack), the method repeatedly computes these spectral summaries from the streaming activations of large language and vision-language models, then feeds the resulting sequence of summaries into small recurrent classifiers (the background chapter covers the Gated Recurrent Unit and Long Short-Term Memory architectures that suit this kind of temporal modeling). The classifier learns to recognize the spectral signature of an impending hallucination or out-of-distribution response, rather than waiting to inspect the produced text.

For compression (RMT-KD), the method inverts the same logic. Spiked covariance theory says a low-rank signal embedded in isotropic noise shows up as a small number of outlier eigenvalues with associated signal directions; BBP theory says which spikes are strong enough to be recoverable. RMT-KD retains the subspaces spanned by those signal directions, projects the network onto them, and then recovers lost accuracy by having the compressed network iteratively distill from its uncompressed self. The criterion for what to keep comes from the statistical theory rather than from a hand-tuned threshold.

The thesis also builds the supporting machinery: definitions of eigenvalues and eigenvectors, spectral decomposition, PCA as a special case (the projection that maximizes preserved variance, equal to the sum of the top-k eigenvalues), and the RMT distributions used as reference baselines.

Why This Matters

Impact on research. The thesis pushes against two common practices in equal measure: treating hallucination detection as a purely behavioral, output-level problem, and treating compression as a hyperparameter-tuning exercise. It argues that both can be reframed as questions about the geometry of internal representations, and that random matrix theory — a mature branch of probability — supplies calibrated reference points rather than heuristics. It also connects to existing threads in the literature such as spectral bias, implicit low-rank structure in convolutional features, and representational similarity analysis.

Real-world applications (as discussed in the paper):

• High-stakes decision support. The thesis repeatedly names healthcare, law, and finance as domains where confidently wrong outputs and out-of-distribution failures carry serious consequences, and where an early, interpretable reliability signal is valuable.

• Edge and resource-constrained deployment. Compact, dense models from RMT-KD are aimed at devices and settings where memory, latency, and energy budgets rule out full-size networks.

• Monitoring deployed models. Because spectral summaries are global and compact, they can be used for continuous monitoring of training health and distributional shifts, not just one-off evaluation.

• Cost and sustainability. The thesis foregrounds inference latency, memory footprint, and energy consumption as first-order concerns for billion-parameter architectures, and frames compression as an environmental as well as an engineering issue.

Industry relevance. Any organization serving LLMs or VLMs at scale faces the same two bills: the cost of running them and the cost of being wrong. A method that produces a dense, hardware-friendly compressed model and a lightweight runtime monitor built from the same spectral statistics is directly relevant to serving infrastructure, model observability, and AI safety tooling. The reliance on hidden activations places both methods in the white-box regime, which is realistic for teams that own their models but restrictive for those that only have API access.

Future Directions

• Publishing the quantitative evaluation. The provided text gives no benchmark names, dataset sizes, model names, compression ratios, or accuracy numbers. A natural next step — and a prerequisite for independent replication — is detailed reporting of these results across the architectures and modalities claimed.

• Extending beyond the architectures tested. EigenTrack is described as evaluated "across diverse architectures" and RMT-KD across natural language processing and computer vision benchmarks; extending both to further modalities and to new recurrent, convolutional, and transformer variants remains open.

• Theory-to-practice gaps in the spectral baselines. The RMT laws used here (Marčenko–Pastur, Wigner, Tracy–Widom, BBP) are asymptotic and assume i.i.d. entries and specific aspect ratios. The thesis itself notes that deep networks rarely behave like pure noise; characterizing when the baselines remain valid for real activations is an open theoretical question.

• Unifying detection and compression into one loop. The thesis explicitly frames reliability and efficiency as two sides of the same spectral geometry. Whether compression changes the hallucination-detection signal — and whether detection signals can guide compression — is the obvious combined research direction.

• Access assumptions. Because both methods read hidden activations, extending the approach to black-box or grey-box settings, where only outputs or partial internals are available, would broaden applicability considerably.

Note: the thesis states that Chapter 6 summarizes key contributions, outlines limitations, and suggests future research directions, but that chapter is not part of the content provided here, so the authors' own specific list of limitations and next steps is not available.

Target Audience

This thesis is most useful to graduate students and researchers in machine learning who want a rigorous, self-contained treatment of random matrix theory applied to neural network internals, and to practitioners working on LLM/VLM reliability or model compression who are willing to read past the mathematics to the methods. Readers with a linear algebra and probability background will get the most from Chapters 2 and 3; readers focused on deployment will care most about the EigenTrack and RMT-KD chapters. Those looking for an immediately reproducible empirical comparison, however, will not find the experimental numbers in the portion of the text available here.

Authors’ abstract

Large language models and deep neural networks achieve strong performance but suffer from reliability issues and high computational cost. This thesis proposes a unified framework based on spectral geometry and random matrix theory to address both problems by analyzing the eigenvalue structure of hidden activations. The first contribution, EigenTrack, is a real-time method for detecting hallucinations and out-of-distribution behavior in language and vision-language models using spectral features and their temporal dynamics. The second contribution, RMT-KD, is a principled compression method that identifies informative spectral components and applies iterative knowledge distillation to produce compact and efficient models while preserving accuracy. Together, these results show that spectral statistics provide interpretable and robust signals for monitoring uncertainty and guiding compression in large-scale neural networks.