Gluing Local Contexts into Global Meaning: A Sheaf-Theoretic Decomposition of Transformer Representations

AI summary

Overview

Research area: Mechanistic interpretability of large language models, sitting at the intersection of natural language processing and algebraic topology (cellular sheaf theory, Hodge theory, spectral graph methods).

Technical level: Advanced. The paper assumes fluency in linear algebra (generalized eigenproblems), graph Laplacians, and basic sheaf cohomology, and it builds a running argument from spectral graph theory into transformer activation analysis. The prose is dense and equation-heavy; readers without a topology background will need to read the definitions sections closely.

Scope (one sentence): The paper builds a cellular sheaf over a graph of paraphrase pairs so that a single eigen-decomposition of a sheaf Laplacian yields three separate tools — a paraphrase-invariant "content" subspace, a set of high-influence "context" directions, and an architecture-level fragility score — and validates each on frozen models with no training.

What This Paper Is About

Transformer hidden states mix what a sentence means with how it happens to be phrased: "The Eiffel Tower is in Paris" and "Paris is home to the Eiffel Tower" produce different activations despite identical content. Existing interpretability tools either decompose a single activation into features, test whether a pre-chosen attribute is linearly decodable, or compare two activation spaces wholesale — none of them directly tests whether a representation keeps the same content when only the phrasing changes. The authors turn each paraphrase pair into a "local witness" that two activations should agree, glue these witnesses into a global object (a cellular sheaf), and read off the part that survives gluing (content) versus the part that resists it (phrasing-dependent variation).

Key Contributions

1. Content from the kernel. The kernel of the sheaf Laplacian, at only 20 dimensions, beats the preserved subspace of the concept erasure method LEACE at full hidden dimension on held-out CounterFact fact retrieval across eight architectures, by a mean of +17.1 percentage points (85.3% vs. 43.1% on GPT-2). Constraining all paraphrases of a fact jointly adds a further +6.5 pp.

2. Influence from the spectral complement. Ablating the most paraphrase-variable coordinates — a coordinate-wise proxy for the top eigenspace — changes model outputs 5.6× to 26.5× more than ablating control coordinates of matched per-coordinate variability, across nine models from 124M to 13B parameters.

3. Fragility from harmonic mass. Stacking layers into a grid creates b₁ = N(L−1) algebraic 1-cycles; the resulting model-derived harmonic mass varies by nearly three orders of magnitude across six architectures and ranks them by steering fragility (Spearman ρ = 0.90, exact one-sided permutation p = 0.014).

4. One graph, three products. The same paraphrase graph supplies the content subspace, the influence directions, and the fragility measure, with the sheaf construction stated formally (coboundary, sheaf Laplacian, restriction maps via joint PCA / CCA / contrastive Fisher) and a proof that on a single-layer matching graph the construction provably reduces to projected within-pair covariance.

Main Findings

• Sheaf kernel beats concept erasure for content. On hard same-relation retrieval with 150 held-out facts from a 70/30 split of 500 ROME CounterFact facts, sheaf H⁰ at 20 dimensions exceeds LEACE at full hidden dimension on all eight architectures, with gaps from +1.5 pp (OLMo-2) to +42.2 pp (GPT-2). Against a dimensionality-controlled LEACE baseline (top 20 PCs of the LEACE-preserved activations), the gap is +27 to +60 pp (mean +48.0 pp). LEACE-20D itself lands within 6 pp of plain PCA-20, consistent with the algebraic prediction that a rank-1 erasure leaves the top-PC ranking essentially unchanged.

• The kernel choice drives the result; the projection is interchangeable. Ablating the restriction map: PCA-P 81.6%, random orthonormal-P 74.2%, identity-P 77.5%, versus PCA-20-without-kernel 30.5% and Random-20 51.2% (means across eight architectures). The authors conclude that content lives in the kernel of the within-pair difference covariance regardless of P, and that PCA merely reduces the eigenproblem from dimension d to 128 at a cost of at most 7.4 pp.

• Joint paraphrase constraints help, but not through cycles. Replacing paraphrase chains with a ring over all of a fact's paraphrases raises H⁰ by +3.2 to +9.5 pp (mean +6.5 pp) across eight architectures; a clique matches the ring within 0.1 pp. An edge-count-matched acyclic path over six paraphrases recovers the gain (+7.7 pp over pairs), and the ring exceeds that path by only +0.5 pp, significant on none of the eight architectures. The cycle structure therefore feeds the harmonic measure, not the content measure.

• Paraphrase-variable coordinates carry outsized causal influence. Table 3 reports ratios of 18.2× (GPT-2, L6), 14.48× (Phi-2, L8), 19.71× (Mistral-7B, L8), 5.76× (Qwen-2.5-7B, L7), 9.17× (OLMo-2-7B, L16), 9.24× (Gemma-2-9B, L21), 18.2× (Llama-2-7B, L16), 5.62× (Llama-3-8B, L16), and 26.5× (Llama-2-13B, L20), with every 95% bootstrap CI excluding 1. GPT-2 (124M) and Mistral-7B reach near-identical ratios (18.2× and 19.71×) despite a 56× size gap. The authors note the control's total variability is only 0.13 to 0.53 of the H¹ set's, so part of the ratio reflects the larger total variability removed.

• Projecting out the H¹ eigenbasis is destructive; matched random ablation is not. Such projections shift next-token distributions 17.8–398× more than random subspaces of equal dimension on all nine models (median over layers), and collapse fact recall to 0–5%, while matched random ablation leaves recall at baseline on the 7B models.

• Harmonic mass tracks steering fragility. With 30 MRPC pairs, joint-PCA per layer at shared k = 8, and b₁ = 30 at L = 2, the model-derived harmonic mass at L = 2 runs from 0.0001 (Mistral-7B, Qwen-2.5-7B) through 0.001 (Llama-3-8B), 0.033 (Phi-2), 0.036 (GPT-2), to 0.079 (Llama-2-7B), against a 0.25 random-probe reference (b₁/|E|). The three lowest-mass models are the three most robust and the three highest are the three most fragile; the single rank inversion (Qwen-2.5-7B 34.3% vs. Llama-3-8B 36.0% preservation) is within sampling error.

• Steering gains are strongly architecture-dependent. On fragile Llama-2-7B, contrastive sheaf steering on H¹ dimensions preserves 31.0% of facts versus 21.1% for PIXEL, 13.7% for full-style CAA and H⁰-removed, 12.1% for sheaf-PCA H¹ dimensions, and 4.2% for random (McNemar p < 10⁻⁷⁰ vs. random). On robust Mistral-7B, CCA-sheaf leads numerically (41.6%) but is a statistical tie with random (39.4%, p = 0.19). On Llama-3-8B, PIXEL (38.8%) and random (36.0%) lead. On Llama-2-13B all methods cluster tightly (29.6–32.7%). Style saturates at 100% for all methods on all models at α = 0.3, so fact preservation is the differentiator.

• The Fisher restriction map underperforms. The Fisher baseline degrades fact preservation (15.4% vs. 29.8%), which the authors attribute to the extra label-driven variability it injects into the edge space.

• Relation hypergraphs produce real topology but inconclusive signal. A hypergraph with 60 facts, 5 paraphrases each, and 6 relations yields 300 nodes, 360 edges, dim H⁰ = 6, dim H¹ = 66 (Euler χ = −60). Cycle holonomy obstructions are 2.5–3.1, below the Haar-random RMS reference √(2k) = 4, and no architecture deviates significantly from a shuffled-paraphrase null (one-sided p = 0.10–0.33).

Methodology in Plain English

The construction has four moves.

1. Put activations on a graph. Collect pairs of sentences that mean the same thing (from MRPC for building the structure, from CounterFact and PAWS for evaluation) and record the model's mean-pooled hidden state for each sentence. Each sentence is a node; each paraphrase pair is an edge.

2. Project pairs into a small shared space. A single matrix P (learned by joint PCA, CCA, or a contrastive Fisher discriminant) compresses the full activation (thousands of dimensions, up to 4096) into a comparison space of k = 128 dimensions, chosen because it captures ≥98% of paired variance. On each edge, the "coboundary" simply records the difference between the two projected activations — a number that should be zero if the pair genuinely means the same thing to the model.

3. Split the disagreement spectrum. Build the 128×128 second-moment matrix of these edge differences (the sheaf Laplacian) and eigendecompose it. The bottom 20 eigenvectors (the kernel, H⁰) are directions along which paraphrases agree, so they should carry content; the top 20 (called "H¹ directions" as a mnemonic) are directions where paraphrases disagree most, so they should carry phrasing.

4. Stack layers to create cycles. A single-layer paraphrase graph has no loops, so the topological machinery collapses to ordinary within-pair covariance (the paper proves this as Proposition 3.1). Stacking sentences across L layers creates a grid with b₁ = N(L−1) square cycles. Each square asks whether the layer-l → l+1 transformation commutes with paraphrase equivalence. Where it does not, a nonzero "harmonic" component appears — the fraction of the network's layer-transition signal that cannot be written as a gradient of any node potential. Because the vertical edges use a fitted orthogonal Procrustes transport R rather than the identity, this mass is genuinely model-derived rather than an artifact.

Validation design. Everything runs on frozen pretrained models with no training. Functional tests ablate coordinates or project out subspaces and measure how much the output changes. Steering tests use CounterFact with 1000 facts per model and α = 0.3‖h̄‖. Retrieval uses 20-dimensional subspaces on a held-out 30% split. Statistics: 1000 bootstrap resamples, McNemar's test for paired comparisons, Cohen's d for effect sizes.

Why This Matters

Impact on research. The paper reframes a vague question — "does this representation encode meaning or wording?" — as a concrete linear-algebraic test that requires no training labels and no per-model tuning, and it does so with a spectral step whose cost is fixed at k = 128 regardless of whether the model has 124M or 13B parameters. It also offers something interpretability notably lacks: a per-architecture predictor of which regime a model is in before any steering is attempted, addressing the documented pattern that steering methods help some models and degrade others. The finding that the restriction map P is largely interchangeable — random and identity maps recover most of the gain — is a strong negative result about how much machinery the method actually needs.

Real-world applications (each a plausible extension, not a claim tested in the paper):

• Safer activation steering. Using the H⁰ subspace as a "do not disturb" region so that style, tone, or safety edits do not silently erase factual content.
• Model selection and deployment triage. Computing a cheap harmonic-mass score on a candidate model to flag whether intervenability work will be easy or fragile before committing engineering effort.
• Continual-update and editing pipelines. Knowledge-editing and unlearning procedures could target coordinates outside the content subspace to reduce collateral fact loss.
• Paraphrase-robust auditing. Checking whether a model's internal representation of a claim stays stable under rewording, as a probe for brittle or shortcut-dependent knowledge.

Industry relevance. The method is training-free, closed-form, and cheap to run, and the spectral step's cost does not grow with model size — the profile of a diagnostic that could realistically sit inside a model-release or evaluation pipeline. The architecture-dependence finding is directly commercially relevant: a steering technique validated on one model family may transfer poorly to another, and this paper supplies a way to predict that.

Future Directions

• Whether the content result holds above 13B parameters. The evaluation spans 124M to 13B; the paper does not report results at larger scales, and the GPT-2 result (+42.2 pp, the largest gap) is attributed to smaller residual streams concentrating paraphrase content into lower-dimensional shared subspaces, suggesting the effect size may shrink with scale.
• Making the cycle-bearing constructions predictive. The depth-induced grid predicts fragility well (ρ = 0.90), but the relation-induced hypergraph does not: holonomy obstructions sit below the Haar-random reference and no architecture differs significantly from a shuffled null. Reconciling these two topological signals is open.
• Filling 2-simplices. Filling each square with two triangles forces b₁ = b₂ = 0 and makes the harmonic mass identically zero; only the coexact (curl) fraction survives, and the paper reports that it preserves the architecture split. Whether that coexact fraction is a viable replacement diagnostic is left open.
• Heterogeneous restriction maps. Per-node maps P_v with Procrustes transports Q_uv break the reduction to projected within-pair covariance (Remark 3.2); the authors note a node-space operationalization of H¹ did not transfer to retrieval or steering, and point to closed-form learned restriction maps as complementary future work.
• Beyond paraphrase equivalence. The framework applies to any equivalence relation on contexts (translations, logical entailments, format-preserving rewrites); the paper only instantiates it on paraphrases.

Target Audience

Researchers and graduate students in mechanistic interpretability who already know their way around transformer internals and want a mathematically principled alternative to feature decomposition; applied mathematicians and spectral-graph-theory researchers looking for a concrete domain where sheaf cohomology earns its keep; and safety/alignment engineers who work on activation steering and need a way to predict which models will tolerate intervention. Readers seeking an accessible introduction to either transformers or sheaf theory will find the paper assumes too much of both — the definitions arrive quickly and the experiments are reported in dense tables.

Note on completeness: the supplied text ends mid-sentence in Section 5.4 (the Llama-2-13B discussion). Sections after that point — including any limitations, broader-impact discussion, and conclusion, and the appendices referenced throughout (A.2, A.3, B.3, B.5–B.7, B.9, B.12, B.13) — were not included in the content provided, so their contents are not summarized here.

Authors’ abstract

Existing interpretability tools do not test whether a representation keeps the same content when only the phrasing changes. We treat each paraphrase pair as a local witness that two activations should agree and build a cellular sheaf over the paraphrase graph; its Laplacian $L_\mathcal{F}$ answers the question on a frozen model with no training. The kernel $H^0$ is a content subspace: at 20 dimensions it beats the preserved subspace of the concept eraser LEACE on held-out CounterFact fact retrieval across eight architectures by a mean $+17.1$\,pp, and constraining all paraphrases of a fact jointly adds a further $+6.5$\,pp. The spectral complement concentrates causal influence: ablating the most paraphrase-variable coordinates, a coordinate-wise proxy for it, changes model outputs $5.6$ to $26.5\times$ more than ablating coordinates of matched per-coordinate variability, across nine models from 124M to 13B parameters. Stacking layers into a grid creates $b_1 = N(L-1)$ algebraic 1-cycles, and the resulting model-derived harmonic mass varies by nearly three orders of magnitude across six architectures and ranks them as their steering fragility does (Spearman $ρ= 0.90$, exact one-sided permutation $p = 0.014$). The same paraphrase graph yields the content subspace, the influence directions, and the fragility measure. Project page: https://cwru-aism.github.io/gluing-lc-page/