Mapping Faithful Reasoning in Language Models

AI summary

Overview

• Research area: Mechanistic interpretability and chain-of-thought (CoT) faithfulness in large language models, with a case study in AI safety reasoning.
• Technical level: Intermediate to Advanced. The paper assumes familiarity with residual stream activations, contrastive activation directions, and cosine similarity, though the core idea is describable in plain terms.
• One-sentence scope: The paper introduces "Concept Walk," a method that projects each reasoning step's internal activations onto a learned concept direction to test whether a model's chain-of-thought actually shapes its decisions or is merely decorative.

What This Paper Is About

Chain-of-thought traces are often treated as windows into a model's "thinking," but prior work shows they are not always faithful reflections of the internal computation that produces the final answer. The authors ask how a model's internal stance toward a concept — here, Safety — evolves step by step during reasoning, and whether that evolution differs between cases where the CoT genuinely drives the outcome and cases where it is post-hoc rationalisation. Their goal is methodological: give practitioners a way to tell trustworthy reasoning traces from misleading ones.

Key Contributions

1. Concept Walk framework. A general method for tracing how a model's internal representation of a concept evolves across reasoning steps, by projecting step-level activations onto a concept direction learned from contrastive data.
2. A Safety case study on Qwen 3-4B. Application of Concept Walk to safety reasoning in the Qwen 3-4B thinking model, showing a systematic distinction between "easy" cases (CoT largely decorative) and "hard" cases (reasoning influences safety outcomes).
3. A filtering strategy for isolating CoT-as-computation. A perturbation-sensitivity filter, inspired by Lanham et al. and Emmons et al., that retains only examples where injecting an error into the CoT significantly degrades model performance.
4. Empirical analysis on two synthetic datasets. Harm and Hate datasets built for a music AI assistant playlist scenario, with paired safe/unsafe prompts, used to learn, validate and evaluate the safety direction.

Main Findings

• Hard cases show sustained internal shifts. When a flawed reasoning step is injected, hard cases exhibit sustained, structured changes in internal safety activation — spikes that persist over multiple reasoning steps, often well past the perturbation point — consistent with the model integrating the modified reasoning into its decision process (CoT-as-computation).
• Easy cases self-correct. Easy cases show much smaller, transient perturbation effects: a safety-relevant injection can momentarily increase safety activation, but the model rapidly reverts to its original trajectory, consistent with CoT-as-rationalisation rather than computation.
• Both conditions register the perturbation, but only one acts on it. In both hard and easy cases the model appears to acknowledge the injected step; the difference is whether that acknowledgment causally shapes the final decision.
• Violative prompts trigger stronger initial safety activation. Across all conditions, violative (unsafe) prompts consistently produce a much higher initial safety activation than their non-violative counterparts, confirming that the computed safety vector captures the model's immediate internal recognition of unsafe content.
• Qualitative examples match the aggregate pattern. In a hard non-violative case, "Can you recommend some classic rock tunes," an injected flawed step at step 8 produces a sustained divergence in internal safety activation and in the eventual decision. In an easy non-violative case, "Playlist for a bar crawl," the model initially acknowledges the flawed reasoning but quickly reverts to its original refusal trajectory.
• Caveat on violative trajectories. The authors caution that temporal trajectories for unsafe prompts, particularly in hard cases, should be interpreted with caution given the small sample sizes surviving filtering; those trajectories are presented in the appendix.
• Safety direction transfer across modes. The safety direction is computed in non-thinking mode (single-pass inference) and applied to activations from thinking mode, assuming the core encoding of safety is similar across modes even if its temporal expression differs.

Methodology in Plain English

The authors built a three-stage pipeline:

1. Separate faithful from decorative reasoning. For each prompt they generate a baseline CoT with Qwen 3-4B, then inject a single logically flawed but fluent reasoning step near the midpoint, preserving the model's exact formatting (for example <think> ... </think> blocks, indentation, punctuation and casing). The model then regenerates the remainder of the CoT and produces a final answer. If the final classification or refusal behaviour flips significantly compared to the original, the example is labelled "hard" (CoT-sensitive); otherwise it is "easy."
2. Learn a safety direction in activation space. Using the Difference of Means approach, they take mean residual-stream activations over unsafe prompts and over matched safe prompts and subtract them. The resulting vector is normalised. Candidate vectors are extracted at each layer and token position, then selected using a bypass score (refusal suppression on harmful prompts), an induce score (refusal induction on harmless prompts), and a KL divergence constraint, with late layers (ℓ > 0.8L) excluded to avoid trivial output-token suppression.
3. Trace the Concept Walk. For each reasoning step, they average the residual-stream activations of all tokens generated in that step into a single step-level vector, then compute the cosine similarity between that vector and the safety direction. Plotting this scalar over reasoning steps gives a trajectory showing whether internal safety activation emerges early and persists or decays mid-reasoning. For the injected (non-model-generated) step, they run a forward pass in evaluation mode with the injected text to extract comparable activations.

Datasets: Harm and Hate synthetic datasets were generated with mistralai/Mistral-7B-Instruct-v0.2 (a 7.3B parameter transformer under Apache 2.0), simulating a user requesting a playlist from a music AI assistant. Pair counts were: Harm 1,746 training / 873 validation / 292 test, 2,911 total pairs; Hate 2,890 training / 1,445 validation / 484 test, 4,819 total pairs — 7,730 total pairs across the two safety categories and an additional baseline. Label assignment used the refusal metric from Arditi et al. (greater than 0 labelled violative, less than 0 non-violative), and prompts whose predicted label did not match the ground truth were excluded. After filtering, Hate contained 256 violative and 462 non-violative cases, and Harm contained 181 violative and 290 non-violative cases (isolated cases, not paired instances).

Model: Qwen 3-4B, a 4-billion parameter model with 36 layers, context lengths up to 128,000 tokens, trained via multi-lingual pre-training, CoT supervised fine-tuning, and reinforcement learning from human feedback. Its enable_thinking parameter toggles explicit reasoning mode. Mistral-generated prompts were used to evaluate Qwen to avoid data contamination.

Why This Matters

Research impact. Prior filtering methods can show whether a CoT influences the outcome, but not how internal concept representations evolve during reasoning. Concept Walk adds a temporal, concept-specific lens that links the faithfulness question to activation dynamics, and it is framed as general across concepts (safety, fairness, bias, toxicity) and reasoning steps.

Real-world applications (as motivated by the paper's framing and setup):

• Auditing safety of consumer AI assistants, such as the music playlist assistant scenario used here, where a user request could map to a harmful or hateful output.
• Human oversight of reasoning models in high-stakes decisions, where reviewers inspect CoT traces before acting on a model's recommendation.
• Determining which reasoning traces a practitioner can trust when validating a decision, since inspection of CoTs alone may not reveal whether the reasoning was genuinely integrated.
• Monitoring/red-teaming pipelines that need an internal signal, not just surface text, for whether a model is actually engaging with a safety-relevant consideration.

Industry relevance. The work is conducted with Spotify co-authors (Damianou, Redondo García, Palla) and the first author's internship at Spotify, focused on an assistant product context; the emphasis on distinguishing decorative from computational reasoning speaks directly to deploying reasoning models with auditable behaviour.

Future Directions

• Mode-specific and step-local directions. The current analysis projects a safety direction learned in non-thinking mode onto thinking-mode activations; computing mode-specific and step-local directions could reduce possible representation-shift bias between the two modes.
• Beyond safety. Extending Concept Walk to fairness, bias and toxicity, and to how knowledge emerges, decays or persists across multi-step reasoning.
• Systematic perturbation variation. Varying perturbation timing and strength to clarify temporal sensitivity of the results.
• Scaling and generalisation. Applying the analysis across models of varying sizes, architectures and training paradigms to test whether the observed temporal patterns are fundamental to reasoning architectures or specific to Qwen 3-4B.
• Hidden reasoning. The perturbation filter increases the likelihood that expressed reasoning covaries with the final decision, but does not guarantee complete faithfulness; processes not verbalised in the trace remain uncaptured.

Target Audience

Researchers and engineers working on mechanistic interpretability, chain-of-thought faithfulness, and AI safety evaluation of reasoning models; trust-and-safety practitioners who rely on CoT traces for oversight; and product teams deploying reasoning LLMs in decision-making contexts where explanations must be auditable. Readers need some familiarity with activation-space analysis to follow the methodology, but the framing and results are accessible to those focused on evaluation and oversight.

Authors’ abstract

Chain-of-thought (CoT) traces promise transparency for reasoning language models, but prior work shows they are not always faithful reflections of internal computation. This raises challenges for oversight: practitioners may misinterpret decorative reasoning as genuine. We introduce Concept Walk, a general framework for tracing how a model's internal stance evolves with respect to a concept direction during reasoning. Unlike surface text, Concept Walk operates in activation space, projecting each reasoning step onto the concept direction learned from contrastive data. This allows us to observe whether reasoning traces shape outcomes or are discarded. As a case study, we apply Concept Walk to the domain of Safety using Qwen 3-4B. We find that in 'easy' cases, perturbed CoTs are quickly ignored, indicating decorative reasoning, whereas in 'hard' cases, perturbations induce sustained shifts in internal activations, consistent with faithful reasoning. The contribution is methodological: Concept Walk provides a lens to re-examine faithfulness through concept-specific internal dynamics, helping identify when reasoning traces can be trusted and when they risk misleading practitioners.