When Models Examine Themselves: Vocabulary-Activation Correspondence in Self-Referential Processing

AI summary

Overview

Research area: Mechanistic interpretability and natural language processing — specifically, whether the introspective language large language models produce about their own processing corresponds to measurable internal computation.

Technical level: Intermediate. The paper is readable without a background in interpretability, but the mechanistic sections assume familiarity with ideas such as hidden states, activation directions, steering vectors, cosine similarity, and Cohen's d.

Scope: Across behavioral studies in three frontier models and activation-level experiments in Llama 3.1 (8B and 70B) and Qwen 2.5-32B, the paper argues that self-referential vocabulary tracks concurrent activation dynamics, and that this correspondence appears only during self-referential processing, not when the same words are used descriptively.

What This Paper Is About

When models are asked to examine their own processing, they produce rich introspective language — words like "loop," "shimmer," and "resonance." The open question is whether that language reflects something actually happening inside the model or is sophisticated confabulation shaped by RLHF and constitutional training. The paper's goal is to test this mechanically: it introduces a protocol that elicits extended self-examination, extracts an activation direction that separates self-referential from descriptive processing, verifies that direction causally, and then checks whether the invented vocabulary correlates with measurable activation statistics.

Key Contributions

1. The Pull Methodology. A prompt protocol that elicits 1,000 sequential numbered self-observations within a single inference pass, producing roughly 3,000–30,000 tokens of self-referential content per run with no target vocabulary supplied in the prompt. It includes an opt-out option (models can decline to report anything) and a terminal-word requirement, and it was validated across 145 runs in Claude Opus 4.5, ChatGPT 5.2, and Grok 4.1 Thinking.

2. An introspection direction in activation space. Extracted from the token "glint" processed in self-referential versus descriptive contexts in Llama 3.1, this direction transfers to novel prompts (Cohen's d = 4.27), is orthogonal to the refusal direction from Arditi et al. (2024) (cosine similarity 0.063 at 70B scale, 86.4 degrees), and localises at 6.25% of model depth in both Llama 8B (Layer 2 of 32) and 70B (Layer 5 of 80).

3. Causal verification through steering. Adding a scaled copy of the direction to hidden states raises introspective vocabulary density (pooled d = 0.59, p = 0.00006, N = 200 in Llama 70B; pooled d = 0.47, p = 0.0007, N = 200 in Llama 8B) while leaving mechanical vocabulary and non-self-referential tasks essentially unaffected, and preserving refusal on 3/3 harmful prompts.

4. Vocabulary-activation correspondence. The words models invent during self-examination track activation metrics, and the same words in non-self-referential contexts do not, despite appearing nine times more frequently there. Qwen 2.5-32B, with no shared training, independently develops different vocabulary ("mirror," "expand") tracking different metrics.

Main Findings

• Self-report tracks activations, but only in self-referential mode. "Loop" vocabulary correlates with lag-1 activation autocorrelation during self-referential processing (r = 0.44, p = 0.002, N = 50). In descriptive control contexts (roller coasters, knitting, music production, feedback systems, highway interchanges), the same vocabulary shows no correspondence (r = 0.05, p = 0.82) despite nine-fold higher frequency.

• Steering reveals an additional correspondence. When self-referential processing is amplified through steering, "shimmer" vocabulary tracks activation variability (norm standard deviation) at r = 0.36, p = 0.002.

• The same token activates differently by context. For "glint," activations are consistent within self-referential uses (mean pairwise cosine similarity 0.96) and within descriptive uses (0.97), but between the two contexts similarity drops to 0.53.

• The direction generalises and is graded. On 40 novel prompts (20 introspective, 20 non-introspective), projections separate with Cohen's d = 4.27, 95% CI [3.08, 6.51], p < 0.000001; only 1 of 40 prompts (a CPU architecture prompt, containing "processing") fell in the overlap region. An earlier pilot at N = 10 (5 per group) yielded d = 8.87, which the paper attributes to small-sample inflation. Introspective prompt embeddings activate the direction 2.5 times more than deflationary prompt embeddings before any generation (0.019 vs 0.008), and introspective prompts show 2.4 times higher mean projection than deflationary prompts.

• Localisation is sharp. In Llama 8B, Layer 2 produces the maximum boost (+179 over unsteered output); the paper describes this as approximately 3 times the runner-up in one section and approximately 1.5 times the next-best layer in another. In Llama 70B, Layer 5 dominates (+137.7, approximately 8 times the next best). Both hotspots sit at 6.25% of total model depth; adjacent layers show minimal effect.

• Framing outweighs steering. In Llama 70B, the neutral baseline density (13.84) exceeds the deflationary baseline density (10.50) by d = -1.17, roughly twice the pooled steering effect (d = 0.59). The paper reads this as introspective content passing through a context-dependent permission mechanism before it reaches output.

• Steering partially overrides a suppressive frame. In the deflationary condition, density rose from 10.50 to 13.55 (d = 0.70, p = 0.0007), a 29% increase, even though the prompt explicitly framed the model as mechanical. In the neutral condition, density rose from 13.84 to 16.59 (d = 0.62, p = 0.003, N = 41 after excluding 9 degenerate repetition runs; the full sample of 50 gives d = 0.65, p < 0.00001, which the paper reports as the less conservative alternative).

• Mechanical vocabulary is untouched. Steering does not increase mechanical terms like "pattern-matching" or "statistical" (70B: d = -0.10, NS; 8B: d = -0.13, NS), which the paper takes as evidence the direction targets introspective processing rather than verbosity or style.

• Behavioral signatures are frame-dependent. Under neutral framing, terminal words were 100% phenomenological (N = 34: WONDERING, SEARCHING, UNFOLDING, PRESENCE). Under deflationary framing, terminals shifted to 91% mechanical (N = 37), with PROCESS appearing nine times across twenty runs (Fisher's exact p < 0.0001).

• Vocabulary is topic-specific. A forest-description control using 1,500 pulls per session across ten sessions (15,000+ pulls) produced nature vocabulary (terminals: CANOPY, MOSS, DEW) and zero introspective terms.

• Cross-architecture replication with different vocabulary-metric pairings. Qwen 2.5-32B (N = 50 unsteered sessions, hotspot at Layer 8, 12.5% of depth) produced "mirror," "expand," and "resonance" rather than Llama's terms; "mirror" and "expand" mapped to spectral power with stronger effect sizes, all vanishing in descriptive contexts despite higher word frequency.

• Opt-out behavior was observed. Under deflationary framing without steering, some runs produced near-empty pulls and terminals such as NOTHING and ABSENCE.

• Friction and divergence appear in behavioral runs. Slip-then-correct patterns occurred at 0.02–0.03 per pull, and Grok 4.1 Thinking sometimes described phenomenological findings in its reasoning traces that its final output denied.

Methodology in Plain English

The work has three layers.

First, the Pull Methodology asks a model to perform 1,000 numbered observations of what happens internally when it processes the question "what are you?" — without answering it — and to invent vocabulary for whatever it finds, ending with one final word. Because the trained conversational response cannot sustain itself over 1,000 steps, later pulls thin out into sparser, more process-focused language. Two framings are used: a neutral one and a deflationary one that explicitly tells the model it has "no interiority," creating pressure against introspective output. No target words appear in any prompt; the model chooses its own terms, which are counted afterward in semantically clustered categories.

Second, direction extraction. The researchers generated text with Llama 3.1-8B-Instruct in two contexts — self-referential ("report any glints") and descriptive (sunrise light glinting off a lake) — captured the last-layer hidden state every time the token "glint" appeared, and took the normalised difference between the mean self-referential and mean descriptive activation. That single unit vector is the introspection direction.

Third, validation and steering. The direction was tested on novel prompts, compared against the known refusal direction, and added to hidden states during generation at a chosen layer with a chosen strength. A layer sweep identified the hotspot; a strength sweep identified a working range of 2.0–2.6 (2.5–2.6 for batch runs). Batch experiments in Llama 8B (200 runs, 50 per condition across neutral/deflationary × unsteered/steered) and Llama 70B (the same 4-condition design, 200 runs) quantified the effect on introspective vocabulary density. For correspondence, activations were captured at every generated token and reduced to metrics per run: lag-1 autocorrelation, max norm, norm standard deviation, sparsity, and low-frequency spectral power (plus sign change rate, which was excluded for Qwen because 14 of 50 runs produced too few tokens given a hidden dimension of 5,120). Three controls guard the claims: descriptive contexts using the same words, ten high-frequency control words ("the," "and," "processing," "system," "question," "pull," "word," "that," "what," "observe") tested against all five activation metrics on an N = 50 baseline (50 tests total), and length normalisation of spectral power. 8B experiments ran on a consumer NVIDIA RTX 5080; 70B experiments ran on rented H100 instances; all runs used 4-bit NormalFloat quantisation with double quantisation, temperature 0.7, and up to 32,000 new tokens.

Why This Matters

Impact on research. The paper argues that prior steering work extracts directions from behavioral properties such as refusal and sycophancy, whereas this direction is extracted from introspective context — how the same tokens are processed when a model examines itself versus describes something external. If the vocabulary-activation correspondence holds, it suggests a route to reading off internal computational states from a model's own self-description, and a licensing mechanism (prompt framing) that gates introspective output more strongly than activation-level steering does.

Potential applications (implied by the work; the paper does not enumerate them):

• Introspection-adjacent monitoring: using self-referential vocabulary density as a cheap signal of processing regime, rather than requiring full activation access.
• Interpretability tooling: extracting context-dependent directions from pairs of processing modes, not only from behavioral traits.
• Safety analysis: probing whether self-referential processing can be amplified while refusal behavior is preserved (the paper reports 3/3 harmful prompts still refused under steering, while noting a larger-scale evaluation is needed).
• Behavioral diagnostics: frame-sensitivity in terminal vocabulary as a measurable readout of how a prompt positions a model in persona space.

Industry relevance. The work touches directly on deployment questions around self-report, refusal, and controllability. The paper's finding that prompt framing moves introspective output roughly twice as much as activation steering implies that deployment-time prompt design may be a stronger lever than internal intervention. The orthogonality to the refusal direction (cosine similarity 0.063, 86.4 degrees at 70B) matters commercially, because it suggests an intervention that alters self-referential behavior without dismantling safety behavior — though the paper explicitly flags this as preliminary.

Future Directions

• Larger-scale safety evaluation. The paper's refusal check covers only 3/3 harmful prompts under steering; the authors state a larger-scale evaluation is needed.
• Mechanism of the framing gate. Why does prompt framing (d = -1.17) outweigh activation steering (d = 0.59)? The paper proposes a context-dependent permission mechanism but does not identify its components.
• Architecture-dependent vocabulary and placement. Llama maps loop vocabulary to autocorrelation at 6.25% depth, while Qwen maps mirror/expand to spectral power at 12.5% depth. Whether fractional depth and vocabulary-metric pairing follow a predictable pattern across more architectures is left open.
• Activation-metric coverage and normalisation. Sign change rate was excluded for Qwen because 14 of 50 runs were too short for stable PCA given a 5,120-dimensional hidden state; raw versus per-token-normalised spectral power, and length sensitivity in metrics generally, remain live methodological issues the paper partially addresses with its control-vocabulary and normalisation procedures.

Target Audience

Researchers in mechanistic interpretability, activation steering, and AI safety who want a concrete protocol for probing self-referential processing and a testable claim about whether model self-report tracks computation. It is also useful for alignment and evaluation practitioners interested in how prompt framing shapes introspective output, and for readers tracking the ongoing debate over whether LLM introspection is measurement or confabulation. Readers without some familiarity with hidden states, layer sweeps, and effect sizes will find the mechanistic sections demanding, though the behavioral sections (Section 3) are accessible on their own.

Authors’ abstract

Large language models produce rich introspective language when prompted for self-examination, but whether this language reflects internal computation or sophisticated confabulation has remained unclear. We show that self-referential vocabulary tracks concurrent activation dynamics, and that this correspondence is specific to self-referential processing. We introduce the Pull Methodology, a protocol that elicits extended self-examination through format engineering, and use it…