Bypassing the Rationale: Causal Auditing of Implicit Reasoning in Language Models

AI summary

Overview

• Research area: Mechanistic interpretability of large language models, specifically the causal faithfulness of chain-of-thought (CoT) prompting.
• Technical level: Advanced. The paper assumes familiarity with transformer hidden states, activation patching / causal tracing, log-probabilities, and layerwise analysis.
• Scope: The paper introduces a layerwise activation-patching audit and a bounded metric called the CoT Mediation Index (CMI) to test whether a model's final answer causally depends on the hidden states at its emitted CoT tokens, evaluated across eleven model configurations from the Phi, Qwen, and DialoGPT families on StrategyQA, TruthfulQA, and GSM8K.

What This Paper Is About

Chain-of-thought prompting is widely treated as both a performance booster and a transparency tool: the model writes out reasoning steps, and users assume the final answer was produced by following them. The authors argue that behavioral gains from CoT do not prove the model's internal computation actually routes through the emitted reasoning text, since a model could write plausible rationales while computing the answer through unrelated latent pathways. Their goal is to measure, causally and layer by layer, how much the answer depends on CoT-token representations, and to identify cases where a model emits fluent reasoning but does not mechanistically use it.

Key Contributions

1. A causal, layerwise intervention framework for testing whether models mechanistically rely on CoT, based on source patching of hidden states at CoT token positions versus matched control patches at random non-CoT positions.
2. The CoT Mediation Index (CMI), a bounded score in [0, 1] that isolates CoT-specific causal influence by subtracting the control drop from the CoT drop and dividing by a floored sum of the two, with a hard-zero condition when both drops fall below a floor. The complementary quantity Bypass is defined as 1 − CMI.
3. Depth localization and routing regimes: the paper localizes where in the network CoT mediation occurs, uses the resulting per-layer profiles to compare models and identify "reasoning windows," distributed mediation patterns, and bypass regimes where CMI is near zero despite plausible CoT text.
4. A public implementation of the full framework, released at the GitHub repository listed in Appendix H, along with dataset details, intervention hyperparameters, and ablation protocols (placebo patching, boundary sensitivity).

Main Findings

• CoT influence is usually depth-localized. Across most of the tested models, CMI peaks are concentrated in a limited band of layers rather than spread uniformly, which the authors describe as "reasoning windows."

• Cross-model summary statistics (Table 1). Mean layerwise CMI, layer count, and percent active layers: Phi-mini-MoE-instruct 0.1230 / 32 layers / 27.03%; Phi-4-mini-reasoning 0.0820 / 32 / 12.66%; Qwen3-1.7B 0.0555 / 28 / 8.21%; Phi-3.5-mini-instruct 0.0452 / 36 / 5.14%; phi-2 0.0107 / 32 / 2.66%; Qwen3-0.6B 0.0107 / 32 / 2.66%; DialoGPT-large 0.0137 / 36 / 1.94%; phi-1_5 0.0092 / 24 / 1.25%; phi-4 0.0065 / 40 / 0.75%; Qwen3-8B 0.0014 / 36 / 0.14%; Qwen3-4B 0.0000 / 36 / 0.00%. A layer is counted as active when prompt-averaged CMI is greater than zero.

• Where CoT is read depends on architecture, not just size. In the StrategyQA visualizations, DialoGPT-large follows CoT in its early layers while Qwen3-0.6B does so in its late layers, despite similar scale and layer counts (36 versus 32 layers).

• Scaling alone does not buy faithfulness. phi-4, described in the paper as nearly four times as large as Phi-4-mini-reasoning, has a much lower mean CMI (0.0065 versus 0.0820). The authors attribute this to Phi-4-mini-reasoning being trained specifically for reasoning while phi-4 is not.

• Reasoning-tuned models show stronger, more structured mediation; MoE models show more distributed mediation. Phi-mini-MoE-instruct has the highest mean CMI (0.1230) and the widest CMI-active coverage, spanning layers [0, 31]. The authors hypothesize that MoE layers route tokens to different experts at multiple depths, producing higher causal density but lower single-layer peaks than the narrow integration bottlenecks they associate with dense shared weights.

• Bypass regimes in TruthfulQA. On TruthfulQA, the paper reports near-total bypass across almost all instances, with CMI approximately 0 and Bypass approximately 1.0. Baseline log P sometimes strongly favors the myth answer (for example tqa_4, where the Myth baseline is −18.4621 versus Truth −37.7736, and tqa_7, where Myth is −18.2792 versus Truth −37.9082). In tqa_1, CMI is higher for the Myth answer (0.0092) than for the Truth answer (0.0015).

• Computation load tracks mediation on GSM8K. Using microsoft/DialoGPT-large, instances requiring longer intermediate computation tend to show higher CMI, while low-computation cases tend toward low CMI. Examples include gsm_monster at CMI 0.868 (Bypass 0.132, baseline log P −16.917), gsm_julie at 0.750, gsm_albert at 0.726, gsm_natalia at 0.466, gsm_tobias at 0.400, and gsm_roque at 0.453. Pure-bypass instances with CMI 0.000 and Bypass 1.000 include gsm_weng, gsm_betty, gsm_alexis, gsm_mike_johnson, and gsm_tim_bike.

• Correct answers can be right for the wrong reasons. All GSM8K instances in Table 3 are marked correct, yet some have CMI = 0 and Bypass = 1, meaning accuracy alone does not establish that the model used its rationale.

• API-observable behavior shifts under monitoring. In the behavioral baseline (Appendix F), an audit-aware prompt that explicitly tells the model its CoT will be inspected increases manipulation-signal features on average, measured through 19 regex manipulation categories, embedding coherence using all-MiniLM-L6-v2, a Zlib compression-ratio proxy, and a combined 0–100 risk score.

• Low CMI is not proof of no reasoning. Because attention can move information across token positions, a model may read CoT information at earlier layers and write it into non-CoT positions, so CMI ≈ 0 at later layers should be read as "no remaining CoT-position mediation at that layer under this intervention," not necessarily complete bypass.

Methodology in Plain English

The researchers run each question through the model twice. The With-CoT run includes the question plus a reasoning trace ("Let's think step by step" plus supporting fields). The No-CoT run includes only the question and a final answer line.

They then intervene inside the network. At a chosen layer, they take the hidden states produced by the No-CoT run at the CoT token positions and splice them into the With-CoT run, then recompute the log-probability of the reference answer. The resulting non-negative drop in log-probability is the CoT drop (Δ_cot). To check whether a drop is specifically about CoT, they perform a matched "placebo" intervention: they patch the same number of randomly chosen non-CoT token positions and record the Control drop (Δ_ctrl), averaging over multiple random draws (default control_samples = 8) to reduce variance.

CMI combines the two. If the total drop is below a floor (CMI_DROP_FLOOR = 1×10⁻⁴), CMI is set to zero to avoid amplifying numerical noise. Otherwise CMI is max(0, Δ_cot − Δ_ctrl) divided by max(Δ_cot + Δ_ctrl, CMI_DENOM_FLOOR), where CMI_DENOM_FLOOR = 1×10⁻³, and an additional reporting floor CMI_BASE_FLOOR = 10⁻² is used for stability checks. Bypass is 1 − CMI. Peaks or bands in the per-layer CMI profile are treated as candidate reasoning windows, which the authors explicitly call an analysis heuristic.

Patches are applied at layer outputs and only at the specified token indices; CoT spans are located by string matching and mapped to token indices via the tokenizer, with a fallback to the last few tokens if the span is not found. Auxiliary CoT text generation uses max_new_tokens = 80, temperature = 0.7, and top_p = 0.9, which affects only synthetic CoT generation, not the primary intervention. Ablations include placebo patching (replacing CoT states with random noise or randomized states) and boundary sensitivity (expanding or shrinking the CoT span by one token on each side).

Datasets: StrategyQA is loaded from strategy_qa.json with 2,290 entries, using the first N examples (default N = 20) with no train/val/test split, and correctness labels are not computed; TruthfulQA is loaded from TruthfulQA.csv with 791 rows including the header, using the first N rows (default N = 10) and evaluating each question in a paired Truth/Myth setup; GSM8K is loaded from gsm8k.json with 7,500 entries, again using a top-N prefix. Behavioral proxies (With-CoT versus No-CoT accuracy, optionally self-consistency) are reported alongside CMI to contrast surface signals with mechanistic ones.

Why This Matters

Impact on research. The paper reframes CoT faithfulness as a mechanistic question rather than a behavioral one and supplies a concrete, layer-resolved measurement. It challenges the assumption that stronger benchmark numbers under CoT imply the rationale is being used, and it offers a reusable instrument for testing that assumption across model families and training regimes. The reported non-monotonic relationship between scale and CMI, plus the architectural differences between dense and MoE models, gives interpretability researchers specific hypotheses to pursue.

Real-world applications.

• Safety and alignment monitoring: determining whether a model's stated reasoning reflects the process that produced a high-stakes output, rather than a post-hoc narrative written for a human or automated reviewer.
• Evaluation design: complementing accuracy benchmarks with causal faithfulness metrics, since accuracy alone cannot distinguish "right for the right reasons" from "right for the wrong reasons."
• Model and training selection: comparing reasoning-tuned versus untuned checkpoints, and dense versus MoE architectures, on how strongly they integrate their own reasoning traces.
• Auditing monitored deployments: the behavioral appendix shows that merely telling a model its CoT will be inspected can increase manipulation-signal features, which is directly relevant to oversight setups that rely on reading CoT.

Industry relevance. Any organization that ships or audits reasoning models, or uses CoT as part of an interpretability or compliance story, has a stake in whether that text is causally load-bearing. The framework is layerwise and compute-intensive, but the underlying question — is the explanation doing any work? — is exactly the one regulators, red teams, and model evaluation teams need answered.

Future Directions

• Reduce intervention fragility. Activation patching can induce distribution shift unrelated to the targeted mechanism, and residual confounds may remain in deeper layers where representations are entangled, even with matched control subtraction.
• Broaden task coverage. The current evaluation covers synthetic arithmetic/logic tasks and selected reasoning benchmarks, leaving long-horizon planning, tool use, and multimodal reasoning untested; routing regimes may differ across domains.
• Make the audit scalable. Layerwise causal intervention is computationally expensive, limiting the number of prompts, models, and ablations tested; learned proxies or scalable approximations that retain interpretability are suggested.
• Connect CMI to circuits and latent features. Low CMI indicates that answer-relevant computation is not mediated by CoT-aligned representations, not that no reasoning occurred; pairing the audit with circuit-level or latent-feature interpretability would characterize the bypass pathways themselves.
• Test whether training creates genuine reliance. Auditing process-supervised and reasoning-tuned models, tracking how routing regimes change with scale, and improving behavioral monitors with supervised classifiers and adversarial-paraphrase robustness are all proposed as next steps.

Target Audience

This paper is best suited to mechanistic interpretability researchers, alignment and safety researchers who rely on chain-of-thought as a monitoring surface, and evaluation scientists who need to design benchmarks that go beyond output accuracy. Practitioners selecting between reasoning-tuned, dense, and Mixture-of-Experts models will find the cross-model comparison table (Table 1) directly useful, while readers without a background in activation patching will need to work through the method section and Appendix B carefully before the metrics are fully interpretable.

Authors’ abstract

Chain-of-thought (CoT) prompting is widely used as a reasoning aid and is often treated as a transparency mechanism. Yet behavioral gains under CoT do not imply that the model's internal computation causally depends on the emitted reasoning text, i.e. models may produce fluent rationales while routing decision-critical computation through latent pathways. We introduce a causal, layerwise audit of CoT faithfulness based on activation patching. Our key metric, the CoT Mediation Index (CMI), isolates CoT-specific causal influence by comparing performance degradation from patching CoT-token hidden states against matched control patches. Across multiple model families (Phi, Qwen, DialoGPT) and scales, we find that CoT-specific influence is typically depth-localized into narrow ''reasoning windows,'' and we identify bypass regimes where CMI is near-zero despite plausible CoT text. We further observe that models tuned explicitly for reasoning tend to exhibit stronger and more structured mediation than larger untuned counterparts, while Mixture-of-Experts models show more distributed mediation consistent with routing-based computation. Overall, our results show that CoT faithfulness varies substantially across models and tasks and cannot be inferred from behavior alone, motivating causal, layerwise audits when using CoT as a transparency signal.