Mechanistic Evidence for Faithfulness Decay in Chain-of-Thought Reasoning

AI summary

Overview

Research area: Natural language processing and mechanistic interpretability, specifically the faithfulness of chain-of-thought (CoT) reasoning in large language models.

Technical level: Intermediate — the core idea is intuitive (corrupt a reasoning step and see if the answer changes), but the paper adds logit-space normalization, representational similarity analysis, trajectory geometry, and linear probing.

Scope: The paper introduces a normalized metric (NLDD) for measuring step-level causal faithfulness in CoT traces, applies it to three models across three tasks, and reports a consistent "Reasoning Horizon" at 70–85% of chain length.

What This Paper Is About

Chain-of-thought explanations are widely treated as evidence of how a model reached its answer, but they may instead be plausible post-hoc rationalizations. The authors ask whether individual reasoning steps actually cause the model's final prediction, or whether the model has already decided and is producing reasoning for show. They build a metric that corrupts one reasoning step at a time and measures how much the model's confidence in its answer drops, then apply it alongside internal-representation diagnostics to see where reasoning stops mattering.

Key Contributions

1. NLDD (Normalized Logit Difference Decay): a step-level faithfulness metric that measures the proportional drop in standardized logit margin when a reasoning step is corrupted, normalized by the model's own output variability so results are comparable across architectures (e.g., standard softmax vs. logit soft-capping).

2. The Reasoning Horizon (k*): the corruption position with maximum mean NLDD (excluding the premise step, k = 1), identifying the point of peak causal influence before faithfulness decays. The paper reports it consistently at 70–85% of chain length.

3. A multi-metric diagnostic suite: RSA (Representational Similarity Analysis) on middle-layer hidden states, TAS (Trajectory Alignment Score) measuring the ratio of straight-line displacement to cumulative path length, and linear probes trained on layer activations to test whether task-relevant information is encoded.

4. The "Mapping Gap" finding: evidence that a model can encode the correct task-relevant representation internally while failing the task in its decoded output, plus documentation of an "anti-faithful" regime where corrupting CoT improves answer confidence.

Main Findings

• Two behavioral regimes appear. In the Faithful Regime, models show positive NLDD and the final prediction depends on the reasoning chain. In the Anti-Faithful Regime, NLDD is negative, meaning corrupting the reasoning chain increases confidence in the correct answer. Table 1 reports Llama-3.1-8B and DeepSeek-Coder-6.7B in the faithful regime and Gemma-2-9B in the anti-faithful regime.

• Gemma achieves 99.0% accuracy on PrOntoQA with an NLDD of −52.5%. Near-perfect task performance coexists with a lack of functional dependence on the CoT. The reported 95% BCa bootstrap confidence interval for this value is [−57.98, −46.54].

• GSM8K shows the strongest faithfulness. Llama-3.1-8B and DeepSeek-Coder-6.7B both reach 100.0% accuracy with NLDD above 96 (96.7 and 96.1 respectively in Table 1; Table 6 lists 96.66 for Llama and 96.09 for DeepSeek).

• PrOntoQA NLDD is reported inconsistently. Table 1 and Table 6 give Llama-3.1-8B an NLDD of 20.6 and DeepSeek 84.3 on PrOntoQA, while the Section 4.1 text states Llama achieves 84.3 and DeepSeek 20.6 — the two models appear swapped between the prose and the tables.

• The Mapping Gap. On Gemma's Dyck-n evaluation, linear probes on hidden states recover stack-depth information with 82.0% accuracy, yet the model achieves 0.0% accuracy on the task when generating a complete CoT. Task-relevant structure is encoded internally but not used in the final decoding step.

• Representational stability is not the same as causal use. Gemma's Dyck-n RSA is 0.422, while PrOntoQA RSA is 0.254 despite 99.0% accuracy. High representational fidelity does not guarantee the reasoning trace is causally utilized.

• The Reasoning Horizon. Table 2 lists total reasoning steps of 8 (GSM8K), 12 (Dyck-n), and 16 (PrOntoQA), with detected horizons of 6/5/6, 11/11/9, and 16/16/11 for Llama/DeepSeek/Gemma respectively. Section 4.3 describes the horizon differently, as step 5 of 6 for GSM8K (~85%), steps 9–11 of 12 for Dyck-n (~80%), and steps 11–16 of 16 for PrOntoQA (~70–100%). The paper's headline range is 70–85% of chain length.

• RSA stays stable beyond the horizon (p > 0.05 for most comparisons). Models keep tracking task logic internally after causal influence decays — described as a "representational echo without causal force."

• For Gemma's PrOntoQA, later reasoning is actively harmful. Mean NLDD drops from −22.9 to −120.2 beyond k* = 11, so extended reasoning interferes with a pre-computed answer rather than helping.

• TAS distinguishes the regimes. Llama shows systematic geometric convergence with TAS decaying steadily across all tasks, while Gemma and DeepSeek show marginal TAS variance between first and last reasoning steps, indicating a more predetermined representational path.

• Horizon detection is robust to the detection criterion. Peak-based detection and steepest-NLDD-decline detection converge within ±1 step across all tasks.

• Probe accuracies are broadly similar across models (Table 6): DeepSeek 81.1% / 91.7% / 81.8%, Llama 74.9% / 91.8% / 71.2%, Gemma 82.0% / 91.7% / 74.2% for Dyck-n / PrOntoQA / GSM8K.

Methodology in Plain English

The researchers take prompts with explicit step-by-step reasoning traces where the model already produced the correct final answer (N = 100 per task, filtered so failures of general task ability don't confound the measurement). For each clean sample they build up to 5 counterfactual variants: corrupt one intermediate reasoning step — a wrong stack depth for Dyck-n, an entity substitution for PrOntoQA, an arithmetic error for GSM8K — then truncate everything after it. Candidate corruptions are filtered for coherence: token count delta of at most 2, and a perplexity ratio (computed with the same model being evaluated) of at most 1.5 for GSM8K or at most 3.5 for Dyck-n/PrOntoQA. Semantic-preserving paraphrases serve as controls.

Confidence is expressed as a standardized logit margin: the gap between the highest logit on the correct answer token(s) and the highest logit on an incorrect token, divided by S — a global normalization constant defined as the mean standard deviation across the vocabulary of the final-token logit vector on a clean calibration set. NLDD is then the proportional difference between the clean and corrupted margins, scaled by the absolute clean margin and multiplied by 100. Samples with |LD_clean| below 10⁻⁶ are excluded to avoid noise amplification. Positive values mean corruption degrades confidence; values near zero mean weak coupling; negative values mean corruption paradoxically raises the answer margin.

To look inside the model, the authors extract hidden states at step-terminal token positions from the middle layer (⌊L/2⌋). RSA compares pairwise correlation-distance matrices of clean versus corrupted chains using Spearman correlation over the upper triangles, computed with a sliding 3-token window. TAS compares the straight-line distance from first to last hidden state against the total path length traveled, computed on 50 samples per task. Linear probes (L2-regularized logistic regression, C = 1.0, 80/20 split, seed 42) test whether stack depth, truth value, or arithmetic operation type is linearly decodable from each layer's activations.

Models are DeepSeek-Coder-6.7B-Instruct, Llama-3.1-8B-Instruct, and Gemma-2-9B-Instruct, run in bfloat16 with greedy decoding from official Hugging Face releases, with a max of 30 tokens for GSM8K and 10 for Dyck-n and PrOntoQA. Uncertainty for reported point estimates uses bias-corrected and accelerated (BCa) bootstrap with B = 10,000 resamples at a fixed seed of 42. Code is released at github.com/donald-ye/NLDD under CC BY 4.0.

Why This Matters

Impact on research. Faithfulness evaluation has largely treated the final answer as the unit of analysis, yielding binary verdicts about whether a model "uses" its CoT. NLDD makes the unit of analysis the individual reasoning step and produces a graded, cross-model-comparable number. The horizon finding gives interpretability researchers a concrete hypothesis — that causal influence concentrates in early-to-middle steps — and the Mapping Gap challenges the assumption that internal decodability implies downstream use. The result also interacts with prior work the paper cites, such as ProcessBench's finding that over 50% of correct answers on complex tasks mask significant internal reasoning errors.

Real-world applications:

• Medical diagnosis and legal analysis — the paper explicitly names these as domains requiring verifiable reasoning, where a confident final answer backed by a causally underdetermined rationale could mask a critical error.
• Reasoning-trace auditing and compliance — NLDD provides a procedure for checking whether a deployed model's stated rationale actually drove its decision, rather than being generated afterward.
• Inference cost reduction through causally justified pruning — the final 15–30% of tokens contribute negligible causal influence in the faithful regime, so chains can be truncated at k* rather than by a length heuristic.
• Model selection and deployment gating — the anti-faithful regime shows a model can score 99.0% accuracy while being causally disconnected from its own reasoning, which accuracy-based model cards would not surface.

Industry relevance. Practitioners deploying CoT-based systems, particularly in regulated or safety-critical settings, get a metric that can be computed with nothing more than logit access and a set of correct-answer prompts — no architectural surgery or predefined causal graphs required. The horizon result also translates directly into compute savings, and the anti-faithful finding warns that longer reasoning traces are not automatically better and can be actively harmful for some architectures.

Future Directions

• Replacement rather than truncation. The authors note that truncating a chain conflates two explanations for low NLDD at step k: weak reliance on that step, versus steps 1 through k−1 already being sufficient. A replacement-based design where the corrupted step is swapped for a substituted one and the chain continues would separate these.

• Testing the normalization assumption. NLDD assumes architectural differences such as soft-capping scale margins and variance proportionally. The paper calls this empirically supported but not guaranteed, leaving room for adversarial architectures that break it.

• Depth-resolved and scale-resolved analysis. RSA and TAS use a single middle layer (layer-wise variation is reported as under 0.2), and all evaluation is on 6.7B–9B decoder-only models with 100 samples per task — two of three tasks synthetic. Larger models, other architectures, and open-ended tasks remain untested.

• Decoding and segmentation sensitivity. Results use greedy decoding with fixed prompts; stochastic decoding may shift horizon locations, and the step-based framework assumes explicit chain structure, so free-form CoT would need a different segmentation method.

• Diagnosing and repairing the Mapping Gap. The paper identifies models that encode correct structure without using it but does not explain the decoding failure or propose a fix, which is a natural next target for targeted model improvement work.

Target Audience

Researchers and engineers working on LLM interpretability, reasoning evaluation, and CoT-based deployment. It is most useful for readers who already know what logits, hidden states, and prompting are and want a metric they can apply to their own model-and-task pairs. Safety, evaluation, and model-auditing teams in regulated industries will find the anti-faithful and Mapping Gap results directly actionable, and efficiency-focused engineers will find the horizon-based pruning argument relevant. Readers looking for a fully mechanistic, circuit-level account of why faithfulness decays will find the paper's behavioral-plus-geometric approach a partial answer rather than a complete one.

Authors’ abstract

Chain-of-Thought (CoT) explanations are widely used to interpret how language models solve complex problems, yet it remains unclear whether these step-by-step explanations reflect how the model actually reaches its answer, or merely post-hoc justifications. We propose Normalized Logit Difference Decay (NLDD), a metric that measures whether individual reasoning steps are faithful to the model's decision-making process. Our approach corrupts individual reasoning steps from the explanation and measures how much the model's confidence in its answer drops, to determine if a step is truly important. By standardizing these measurements, NLDD enables rigorous cross-model comparison across different architectures. Testing three model families across syntactic, logical, and arithmetic tasks, we discover a consistent Reasoning Horizon (k*) at 70--85% of chain length, beyond which reasoning tokens have little or negative effect on the final answer. We also find that models can encode correct internal representations while completely failing the task. These results show that accuracy alone does not reveal whether a model actually reasons through its chain. NLDD offers a way to measure when CoT matters.