Geometry of Reason: Spectral Signatures of Valid Mathematical Reasoning

AI summary

Overview

Research area: Mechanistic interpretability and AI safety, specifically the use of spectral graph theory to verify reasoning in large language models.

Technical level: Advanced. The paper assumes familiarity with transformer attention, graph Laplacians, eigendecomposition, and effect-size statistics.

Scope (one sentence): The paper treats each transformer attention matrix as a weighted token graph and shows that four training-free spectral diagnostics, computed from the graph Laplacian and the hidden-state signals on its vertices, separate valid from invalid mathematical proofs across seven models from four architectural families.

What This Paper Is About

The core problem is that when a language model produces a mathematical proof, there is no cheap way to tell whether it genuinely reasoned or merely pattern-matched a plausible-looking argument. Existing options are either output-based (formal provers such as Lean, Coq, or Isabelle, which conflate logical validity with syntactic acceptance and can reject correct proofs over timeouts or missing imports) or learned (classifiers trained on model internals or outputs, which need large labeled datasets and may not transfer across architectures).

The goal is a verification method that requires no training, no sampling, and no labels, and that reads the geometry of attention rather than the wording of the output. The authors hypothesize that valid proofs induce smoother, better-connected attention graphs, while invalid reasoning produces spectral irregularity.

Key Contributions

1. A training-free framework for reasoning validity detection based on spectral analysis of attention graphs, reporting 82.8–85.9% accuracy under nested cross-validation and up to 95.6% with calibrated thresholds (Sections 3 and 4).
2. Cross-architecture universality across seven models from four families, with |d| ≥ 2.09 and p_MW < 10^-47 in all cases, and robustness across all difficulty strata (d ≥ 1.31 for complex proofs).
3. The discovery of "Platonic validity": the spectral method detects logical coherence rather than compiler acceptance, identifying mathematically sound proofs that formal systems reject on technical grounds (Section 5.2).
4. A practical downstream result (reported in the abstract and Section 4.6): using HFER as a reranker in proof search improves Best-of-16 Pass@1 by +4.4–6.6% and matches 98% of the AUC of fully supervised probes with zero labels.

Main Findings

• Valid proofs share a consistent spectral signature. Across all seven models, valid proofs show lower HFER (energy concentrated in low-frequency, smooth spectral modes), higher spectral entropy, higher smoothness, and a higher Fiedler value (better-connected attention graphs). Only the magnitude and the optimal metric vary by architecture.
• Effect sizes are exceptionally large. The best effect is Cohen's d = 3.30 (Phi-3.5-mini, smoothness at Layer 25; p_MW = 4.51 × 10^-66). Qwen2.5-0.5B reaches p_t = 1.43 × 10^-116 with d = 2.93. The weakest model, Mistral-7B-v0.1, still shows d = 2.09 and p_MW = 1.16 × 10^-48.
• Single-threshold classification reaches 85.9–94.9% across the seven models. The abstract summarizes this range as 85–96%. Per-model accuracies: Llama-3.2-1B 93.4% (Fiedler, L0), Llama-3.2-3B 94.9% (HFER, L11), Llama-3.1-8B 94.1% (HFER, L30), Qwen2.5-0.5B 93.2% (entropy, L0), Qwen2.5-7B 89.9% (HFER, L26), Phi-3.5-mini 93.4% (smoothness, L25), Mistral-7B 85.9% (smoothness, L26). The majority-class baseline is approximately 57%.
• The signal tracks logic, not authorship or style. On n = 16 reclaimed (semantically correct but Lean-rejected) proofs versus n = 16 genuinely erroneous model-generated proofs, the Fiedler value at the final layer separates the groups (p = 0.002, d = 1.30; n = 32 total). On n = 40 perturbations of human-valid proofs on Llama-3.2-1B, HFER at the final layer rises from 0.331 (valid) to 0.378 (perturbed), p = 1.93 × 10^-9, d = 1.10.
• "Platonic validity" versus compiler validity. All seven models independently flagged 33–51 compiler-rejected proofs as spectrally valid. A manual audit of 51 reclaimed proofs by two independent raters (Cohen's κ = 0.82) found that 64.8% fail Lean only on technical grounds: semantically valid with minor structural issues 37.3%, environment or missing imports 27.5%, timeout or computational limit 13.7%, incomplete (model admits failure) 13.7%, syntax or version issues 7.8%. The paper's worked example is mathd_numbertheory_961 (theorem 2003 mod 11 = 1, proof norm_num), which Lean rejects on a timeout but which the spectral classifier labels valid and which is mathematically correct.
• Architecture determines which spectral channel encodes reasoning quality. Mistral-7B-v0.1, the only model using Sliding Window Attention, shifts the discriminative feature from HFER to smoothness (d = 2.09, p < 10^-48).
• The signature traces induction-head circuits, per a causal ablation.
• Calibration cost is real but small. Nested cross-validation (5-fold outer / 4-fold inner) gives 82.8–85.9%; a 60/20/20 train/val/test split gives 73.6% (Mistral-7B) to 83.5% (Llama-3B, Qwen-0.5B). Under Benjamini-Hochberg correction at FDR = 0.05 over 160 hypotheses (5 metrics × 32 layers), 156/160 (97.5%) remain significant for Llama-3.1-8B. The most frequently selected configuration under nested CV is mid-to-late layer HFER (6 of 7 models), with Phi-3.5 selecting smoothness at Layer 25.
• Robustness to ablations. Perturbing the optimal threshold by ±10% changes accuracy by less than 2.5%; ±20% yields less than 5% degradation. Accuracy is 100% on IMO/Putnam problems (n = 12) and 93% on AMC/AIME. Accuracy is stable at 87–100% across proof-length quintiles. Raw threshold transfer between models fails (accuracy drops to about 50%), but calibrating for a new model requires only about 50 labeled examples.
• Metric independence is partial. HFER and spectral entropy correlate at r = −0.97 on average, while the Fiedler value is largely independent (r = −0.29 with HFER). At Layer 8 of Llama-3.1-8B, HFER reaches d = 2.39 (95% CI [1.99, 2.83]) while spectral entropy collapses to d = 0.08 (95% CI [0.00, 0.23]). On Mistral-7B, the HFER–entropy correlation drops from r ≈ 0.9 in dense models to r = 0.45.
• The signal survives in informal language, weakly. On Llama-3.2-1B over MATH (N = 227 informal chain-of-thought samples), the effect attenuates from d = 3.02 (formal) to d = 0.78 (informal) but stays significant (p < 10^-3). The optimal metric shifts to the Fiedler value at Layer 14. On balanced data (N = 106), the spectral threshold reaches 68.4% accuracy, beating the 50.0% majority-class baseline and the 74.5% Random Forest on full data (which is not reported for balanced data in Table 3). Full-data numbers: majority class 76.6%, Random Forest 74.5%, spectral threshold 77.1%.
• HFER reranking improves proof search. On Best-of-16 (N = 16, T = 0.7) with Llama-3.1-8B on MiniF2F, HFER reaches 34.2% Pass@1 (AUC 0.962) versus 29.8% for log-probability (AUC 0.979), 30.4% for token entropy (AUC 0.971), 22.4% for random selection, and 37.1% (AUC 0.988) for an ensemble using Z_LP − Z_HFER. The AUC–Pass@1 inversion is attributed to log-probability being blind to "confident hallucinations." On Phi-3.5-mini (d = 3.30), HFER achieves 37.8% versus log-probability's 31.2% (+6.6%), compared with +4.4% on Llama-3.1-8B (d = 3.00).
• Comparison to a supervised probe. Against the probe of Obeso et al. (2026) trained on Llama-3.1-8B Layer 16 hidden states: that probe uses millions of labels and reaches AUC 0.981; a linear probe trained by the authors on 363 labels reaches 0.949; HFER calibrated on 50 labels reaches 0.962; HFER zero-shot with 0 labels reaches 0.923. With 50 calibration examples, HFER achieves 98% of the fully supervised upper bound (91.8% ± 2.4% accuracy).
• Difficulty stratification (Llama-8B). Trivial (1 tactic, n = 107) d = 6.69; simple (2–3 tactics, n = 69) d = 3.29; moderate (4–6 tactics, n = 72) d = 2.09; complex (≥7 tactics, n = 206) d = 1.31.
• Sparsity penalty in mixture-of-experts. Evaluating Qwen-MoE, effect sizes attenuate from d ≈ 3.0 (dense) to d ≈ 1.6 (sparse) on balanced data, though the signal remains significant (p < 10^-10), and the dominant

Authors’ abstract

Verifying whether a language model is genuinely reasoning or pattern-matching remains an open problem: learned verifiers are expensive, and output-based heuristics are brittle. We show that valid mathematical reasoning induces a measurable, training-free spectral signature in transformer attention. By treating each attention matrix as a weighted token graph, we extract four diagnostics: Fiedler value, High-Frequency Energy Ratio (HFER), spectral entropy, and smoothness, that require no learned parameters. Experiments across seven models from four architectural families yield effect sizes up to Cohen's $d = 3.30$ ($p &lt; 10^{-116}$), enabling $85$--$96\%$ single-threshold classification accuracy. Two findings sharpen the interpretation. First, \emph{Platonic validity}: the spectral signal tracks logical coherence rather than compiler acceptance, proofs rejected for timeouts or missing imports are correctly classified as valid, a distinction confirmed by a manual audit ($κ= 0.82$, $n = 51$). Second, \emph{architectural determinism}: Sliding Window Attention shifts the discriminative feature from HFER to smoothness ($d = 2.09$, $p &lt; 10^{-48}$), showing that attention design governs which spectral channel encodes reasoning quality. Causal ablation confirms the signature traces induction-head circuits. The method generalises to informal chain-of-thought ($d = 0.78$, $p &lt; 10^{-3}$), and in proof search, HFER reranking improves Best-of-16 Pass@1 by $+4.4$--$6.6$\%, matching $98\%$ of the AUC of fully supervised probes with zero labels. Spectral graph analysis is a principled, architecture-aware primitive for reasoning verification.