Beneath the Surface of Chains-of-Thought: A Mechanistic Interpretation of Reasoning Operations in LLMs

AI summary

Overview

Research area: Mechanistic interpretability of large language models, specifically the internal representation geometry of chain-of-thought reasoning steps.

Technical level: Advanced. The paper assumes familiarity with hidden-state probing, linear discriminant analysis, PCA, AUROC/AUPRC evaluation, and attention-masking interventions.

Scope: The paper tests whether functionally distinct reasoning operations expressed in a model's chain-of-thought trace (such as decomposition or arithmetic computation) occupy separable, context-dependent directions in the model's hidden representation space.

What This Paper Is About

Chain-of-thought traces mix many different functional steps, for example restating a problem, breaking it into subproblems, recalling a formula, or doing arithmetic. The paper asks whether these textually distinguishable operations also correspond to distinguishable geometric structure inside the model's hidden states, beyond the mere identity of the words being generated. The goal is to determine where in the network that structure appears, how it is distributed across tokens, and whether it depends on preceding reasoning context.

Key Contributions

1. An operation taxonomy for reasoning traces. The authors build a hierarchical taxonomy of reasoning operations from Pólya's four-stage problem-solving framework (How to Solve It, 1945), and select eight recurring operation types for analysis: Extraction, Direct mapping, Decomposition, Recall, Deduction, Algebraic manipulation, Arithmetic computation, and Final answer.

2. Evidence that reasoning operations are linearly separable in hidden states. Using LDA probes fit only on training splits, the authors show one-vs-rest AUROC is consistently high across Qwen2.5-7B, Qwen3-8B, and Gemma4-31B, with separability peaking in middle layers.

3. A set of controls ruling out lexical and positional explanations. Text-only classifiers, lexically matched span comparisons, competing-operation vocabulary subsets, and digit/formula-density restrictions all leave operation-level separability intact, and a PCA-only variant (no LDA) preserves the main result.

4. Causal and error-conditioned analyses of operation representations. Attention-masking interventions show that preceding reasoning context contributes to the operation-aligned representation at a chunk's onset, and probes trained on correct traces still detect operation identity in factual-error spans, though more weakly.

Main Findings

• Separability peaks in middle layers. Middle-token representations yield high peak-layer one-vs-rest AUROC for reasoning operations across Qwen2.5-7B, Qwen3-8B, and Gemma4-31B, with separability strongest in the middle layers and lower at embedding, early, and late depths.

• Hidden states beat text-only baselines. Mean-pooled hidden-state probes outperform the stronger of bag-of-words and TF-IDF logistic regression by 0.041 to 0.097 in macro AUROC and 0.084 to 0.193 in macro AUPRC. In the reported table, Qwen3-8B scores 0.718/0.279 for position-only, 0.718/0.279 for text-only, and 0.937/0.742 for hidden states; Qwen2.5-7B scores 0.708/0.269, 0.854/0.562, and 0.895/0.646; Gemma4-31B scores 0.751/0.311, 0.802/0.466, and 0.899/0.641.

• Operation identity beats broad lexical similarity. In lexically matched comparisons, the median effect favors operation identity over lexical similarity across all eight operations, with confidence intervals excluding zero for five of them.

• Adversarial vocabulary does not override operation identity. On held-out spans containing vocabulary strongly associated with a competing operation, the original probe reaches macro AUROC/AUPRC of 0.919/0.848 for Qwen3-8B and 0.884/0.789 for Qwen2.5-7B.

• Digit and formula density is not the explanation. Restricting evaluation to spans with 50–75% digit or mathematical-token density leaves five operation types retained, with macro AUROC/AUPRC of 0.917/0.789 using mean-pooled representations. All five stay above chance; the smallest margin is for Arithmetic Computation (AUPRC 0.510 vs. 0.204).

• LDA is helpful but not required. With 128 principal components and training-set class-mean directions instead of LDA, macro AUROC/AUPRC remains 0.938/0.716 for Qwen3-8B, 0.914/0.700 for Qwen2.5-7B, and 0.872/0.552 for Gemma4-31B.

• Position alone is weaker than hidden states. Across five position bins, frozen probes still achieve macro AUROC between 0.916 and 0.973 and macro AUPRC between 0.769 and 0.898.

• Generalization beyond the original setup. Llama-3-8B yields macro AUROC/AUPRC of 0.958/0.840 with mean-pooled representations, and Qwen3-8B probes trained on the original tasks transfer without retraining to GPQA-Diamond (0.938/0.764) and MATH-500 (0.948/0.799).

• Signals become span-distributed in middle layers. Intra-span variance of operation-alignment scores is relatively high in early layers, decreases toward middle layers, and rises slightly again near the final layers, indicating that middle-layer separability is not driven by a few isolated cue tokens.

• Identical tokens get operation-dependent representations. Shared surface-token occurrences are largely intermixed in early layers, separate along operation-specific directions in middle-to-late layers, and become more mixed again in the final layer.

• Preceding context causally shapes operation onsets. Masking the preceding 30 tokens reduces the target operation-alignment score at the onset of a reasoning chunk, indicating the onset representation is not formed from chunk-local content alone. Preceding-chunk masking shows a qualitatively similar direction, and random-chunk masking is reported as a supplementary control on a smaller eligible sample.

• Factual errors weaken but do not erase operation geometry. With mean-pooled representations, probes trained only on correct traces reach 0.955/0.877 on factual-error spans versus 0.971/0.901 on operation-matched non-error spans from the same incorrect traces; with middle-token representations the scores are 0.920/0.759 and 0.937/0.808. Deduction and Arithmetic Computation show the clearest reductions, while Recall shows little difference.

• Annotation reliability is measured, not assumed. Seven annotators (including three authors) labeled 84 sampled spans, with three annotators per span. Majority agreement was reached on 81 of 84 spans (96.4%; Fleiss' κ = 0.666), and GPT-5 matched the human majority in 64 cases (76.2%; Cohen's κ = 0.715). Random guessing over eight labels would give 12.5% expected exact agreement.

Methodology in Plain English

The authors take reasoning models and let them generate full solution traces on math and theorem problems, keeping only traces with correct answers (500–700 per model and dataset). A separate model, GPT-5, chops each trace into contiguous spans and tags each span with one of eight operation labels. Long spans are trimmed: spans over 300 tokens are dropped, and spans over 50 tokens are replaced by a 50-token window centered on the highest-entropy token.

For each span, the authors pull the hidden-state vector from every layer. They L2-normalize, reduce to 128 dimensions with PCA, and fit a supervised LDA using the operation labels. Because there are eight classes, the LDA space has at most seven dimensions. From the training data they compute a one-vs-rest "operation vector" for each label: the difference between the mean of that operation's spans and the mean of everything else, normalized. A span's alignment with an operation is just the dot product between its projected representation and that operation vector, and this score is used as a one-vs-rest classifier scored by AUROC. All normalization, PCA, LDA, and direction estimates come from the training split only; the test split is projected without refitting. Class imbalance is limited by sampling at most 300 training and 60 test spans per operation, and matched controls repeat the pipeline with random labels and random token positions.

To check the results are not just word statistics, they compare against text-only classifiers and against tightly matched span pairs. To see how signals are distributed, they compute the variance of token-level alignment scores within each span and track how the same surface token scores under different surrounding operations. To test causation, they re-run the model on the fixed generated sequence while blocking attention from the first token of a chunk to preceding tokens, and measure the change in alignment. Finally, they look at traces with wrong final answers, split spans into those with explicit factual errors and matched spans without, and apply the correct-trace probes unchanged.

Why This Matters

Impact on research. The paper bridges two lines of work that have mostly stayed separate: analyses of the observable structure of reasoning traces (thought anchors, reasoning behaviors, discourse graphs) and analyses of hidden-representation geometry. It shifts the unit of analysis from whole trajectories or answer-correctness signals down to local, recurring functional operations, and it provides a fitted, transferable set of operation vectors rather than only qualitative claims. The transfer results to GPQA-Diamond and MATH-500 without task-specific retraining suggest the structure is not dataset-specific.

Real-world applications (the paper identifies these as unexplored directions rather than demonstrated uses):

• Reasoning failure detection, by checking whether an operation's representation deviates from its expected direction.
• Verification of intermediate reasoning steps during generation.
• Decoding-time control that steers a model toward or away from specific operations.
• Activation-based steering to shape how a model plans versus executes.

Industry relevance. Training pipelines increasingly optimize the reasoning trajectory itself, not just the final answer. Knowing that operation identity is recoverable from middle-layer activations gives practitioners a concrete monitoring signal and a candidate intervention point, and the released code and project materials at https://github.com/naver-ai/beneath-cot make the probes reproducible.

Future Directions

• Scale up annotation validation. Human validation covered 84 spans and focused on operation labels rather than span boundaries. The authors call for larger-scale validation of both labels and boundaries, and note that GPT-5 labels should be treated as approximate rather than as ground truth about latent cognitive states.

• Test beyond math and theorem tasks. All experiments use DAPO-Math-17K and TheoremQA with three main models plus Llama-3-8B. The authors state that the observed geometry may differ in commonsense reasoning, planning, code generation, or interactive tasks, and in models with different architectures or training procedures.

• Move from diagnosis to intervention. The paper is explicitly diagnostic. Whether the learned operation vectors can improve behavior through failure detection, verification, decoding-time control, or activation steering is left open.

• Explain operation-dependent error attenuation. Attenuation under factual errors was clearest for Deduction and Arithmetic Computation and nearly absent for Recall. What makes some operation representations robust to incorrect execution and others not remains unexplained.

Target Audience

Researchers in mechanistic interpretability and LLM reasoning who want to connect observable chain-of-thought structure to internal representation geometry; engineers building reasoning models who need monitoring or steering signals from middle-layer activations; and graduate students with background in linear probing, dimensionality reduction, and evaluation metrics, since the methods are standard but the experimental design is dense. Readers without that background will find the probing pipeline and the statistical controls difficult to follow from the main text alone, as much of the detail sits in the appendices.

Authors’ abstract

Reasoning in large language models unfolds through diverse functional operations, such as problem formulation, goal decomposition, and deduction. Although these operations are explicitly distinguished in text, little is known about how they are geometrically organized in representation spaces. To this end, we investigate whether distinct reasoning operations exhibit corresponding geometric structure in hidden representations. We find that operations are separable in held-out representations, with separability peaking in middle layers, and verify that this structure is not explained by lexical or positional confounds. Across layers, token-wise operation-alignment becomes more distributed over spans, while identical surface tokens are represented differently depending on the operation of its surrounding chunk. Attention-masking interventions further show that operation-aligned representations at chunk onset depend on preceding reasoning context. Consequently, our work demonstrates that language models maintain representational correspondence between linguistic reasoning expressions and their internal geometric structures. Code and project materials are available at https://github.com/naver-ai/beneath-cot.