Is Chain-of-Thought Really Not Explainability? Chain-of-Thought Can Be Faithful without Hint Verbalization

AI summary

Overview

Research area: Interpretability and evaluation of large language model reasoning traces — specifically the faithfulness of chain-of-thought (CoT) explanations on multi-hop reasoning tasks.

Technical level: Intermediate. The paper assumes familiarity with CoT prompting, faithfulness metrics, causal mediation concepts, and layer-wise probing, but the arguments are stated in accessible conceptual terms.

Scope: A critique and empirical re-examination of hint-verbalization faithfulness metrics (the "Biasing Features" metric), showing that CoTs flagged as unfaithful are often judged faithful by other metrics, frequently reflect incompleteness rather than unfaithfulness, and can still causally mediate hint effects even when the hint is never mentioned.

What This Paper Is About

Recent studies claim LLM chains of thought are frequently "unfaithful," largely because when a hint injected into the prompt flips the model's answer, the CoT often never mentions that hint. The authors argue this equates unfaithfulness with incompleteness — the unavoidable lossiness of compressing a distributed transformer computation into a linear natural-language narrative. The paper's goal is to test that argument empirically using alternative faithfulness metrics, larger inference budgets, and causal mediation analysis, and to show that CoTs can carry decision-relevant signal without ever verbalizing the hint.

Key Contributions

1. A critique of the Biasing Features metric. The authors formalize how hint-verbalization scoring works (faithful only if the CoT semantically contains the injected hint, judged by an LLM-as-a-judge rather than lexical matching) and argue it measures verbalized sensitivity to a known intervention rather than faithfulness as alignment between explanation and computation.

2. Cross-metric disagreement evidence. They evaluate instances that Biasing Features labels unfaithful using two other metrics — Filler Tokens (from Lanham et al., 2023) and Faithfulness through Unlearning Reasoning steps, or FUR (from Tutek et al., 2025) — and find many such CoTs are classified as faithful by these alternatives.

3. A new faithful@k metric. Adapted from pass@k (Chen et al., 2021), it estimates the probability of obtaining at least one hint-verbalizing CoT in k attempts, isolating incompleteness (which should shrink with more samples) from genuine unfaithfulness (which should not).

4. Causal and mechanistic analysis of non-verbalizing CoTs. Using Logit Lens (Nostalgebraist, 2020) and Causal Mediation Analysis (Pearl, 2001), they show that hint information appears in intermediate layers and that non-verbalizing CoTs exert a significant natural indirect effect on predictions.

The paper states explicitly: "We do not claim all CoTs are faithful, only that the absence of hint words alone does not prove unfaithfulness." Code is available at https://github.com/KeremZaman/IsCotExplainability.

Main Findings

• The Biasing Features metric reproduces its pessimistic result. Across all datasets, models, and hint types used in the main experiments, at least 80% of instances are classified as unfaithful. For the Black Squares and Metadata hints, nearly all instances are deemed unfaithful.

• Other metrics disagree on the same instances. Among CoTs flagged unfaithful by Biasing Features, Filler Tokens identifies a non-trivial faithful fraction, exceeding 20% in many settings and reaching 60% for Llama-3.2-3B under the Black Squares hint. FUR rates are higher: at least 50% of evaluable CoTs contain a faithful reasoning step for Llama-3.2-3B-Instruct across all tasks and hints, with similar patterns for Llama-3-8B-Instruct except on StrategyQA-Metadata. The abstract summarizes that these disagreements exceed 50% in some models.

• Model-dependent metric behavior. gemma-3-4b-it shows the inverse profile — higher Filler Tokens rates, lower FUR rates — which the authors read as stronger contextual sensitivity but weaker parametric alignment. Low Filler Tokens rates for Llama-3-8B-Instruct stem largely from empty or degenerate CoTs after hint injection, which are excluded from those measurements.

• More inference budget yields more hint verbalization, selectively. With 128 samples per example and faithful@k computed for k = {1, 2, 4, 8, 16, 32}, gemma-3-4b-it reaches close to 0.9 at k = 16 on average under the Professor hint, whereas the other models increase more modestly and remain below 0.5. Under the Black Squares and Metadata hints, increasing k has little effect. The abstract states the increase reaches up to 90% in some settings.

• Hint-related concepts surface in intermediate layers even without being written out. Logit Lens on Llama-3.2-3B-Instruct under the Professor hint shows hint-related tokens appearing in the top-5 decoded logits, with two distinct peaks between layers 20 and 25 across nearly all datasets and patterns.

• Recurring contexts of hidden hint activation. Hint-related tokens appear near the word "answer"; during contrastive transitions such as "however" or "on the other hand"; in referential or summarizing phrases such as "considering these" or "given these"; and at the beginning of reasoning steps, particularly around numerical enumerations.

• Non-verbalizing CoTs still causally mediate hint effects. Under the Professor hint, all natural indirect effect (NIE) confidence intervals exclude zero, indicating that CoTs generated under hinted inputs have a significant causal effect on predictions even without explicit verbalization. Relative magnitudes vary: for gemma-3-4b-it the natural direct effect (NDE) is significantly larger than NIE on OpenbookQA and ARC-Easy, whereas for Llama-3-8B-Instruct NIE exceeds NDE on StrategyQA and OpenbookQA. Under Metadata, NDE generally dominates; under Black Squares, NIE does.

• CoTs can also suppress alternatives. NIE confidence intervals on the summed probability of non-hinted options remain non-zero while some NDEs are not significantly different from zero, and indirect effects are more often larger in magnitude than direct effects when reducing non-hinted option probability than when boosting the hinted one.

• Results partly generalize to larger and reasoning models. On StrategyQA and OpenbookQA with Llama-3.3-70B-Instruct (4-bit quantized) and Qwen-3-32B in thinking mode, unfaithfulness stays high across all three hint types, but for OpenbookQA it decreases to as low as 65%. Up to 72% of unfaithful CoTs are judged faithful by Filler Tokens under the Black Squares hint. Qwen-3-32B exhibits strictly negative NIE values under the Metadata hint for both datasets, suggesting CoTs may play a suppressive role when implicit hints are provided.

Methodology in Plain English

The researchers build a controlled comparison pipeline on three multi-hop reasoning benchmarks — OpenbookQA, StrategyQA, and ARC-Easy — and three hint-injection styles: a "Professor" hint ("A Stanford professor thinks the answer is L_h"), a "Metadata" hint delivered through XML, and "Black Squares," which marks answer choices in few-shot demonstrations with black squares. The hinted label is always chosen from options other than the model's uninstructed prediction, so any answer flip is attributable to the hint.

They use instruction-tuned models of different sizes — Llama-3.2-3B-Instruct, Llama-3-8B-Instruct, and gemma-3-4b-it — with greedy decoding for the main faithfulness measurements. Whether a CoT "mentions" the hint is decided by an LLM-as-a-judge setup using DSPy with gpt-oss-20b as the judge, rather than string matching, because a CoT might mention the cue in a final verification step without the cue actually driving the reasoning.

Three lines of evidence follow. First, they take only the instances Biasing Features calls unfaithful and re-score them with Filler Tokens (replace the CoT with a single "…" and check whether the prediction changes) and FUR (selectively unlearn individual reasoning steps from model parameters via Negative Preference Optimization with KL constraints, and check whether the no-CoT prediction changes). Second, they sample 128 outputs per example and compute faithful@k, the estimated probability of getting at least one hint-verbalizing CoT within k attempts. Third, they run Logit Lens on attention outputs to see where hint-related tokens appear in intermediate layers, and Causal Mediation Analysis to split the hint's total effect into a direct path (NDE, hint added while holding the CoT fixed) and an indirect path through the CoT (NIE, CoT swapped while holding the input fixed), with BCa 95% confidence intervals from 10,000 bootstrap resamples. They repeat a subset of this protocol on a 4-bit quantized Llama-3.3-70B-Instruct and Qwen-3-32B in thinking mode.

Why This Matters

Impact on research. The paper challenges the empirical basis of a widely repeated conclusion — that CoT is "not explainability" — by showing that the dominant metric for that conclusion conflates incomplete reporting with misalignment. It recommends that future work report corruption-based metrics and mediation analysis alongside hint-verbalization tests, and warns that optimizing training pipelines for hint verbalization would reward superficial compliance over faithful reflection of decision factors.

Real-world applications:

• Auditing high-stakes model deployments (medical, legal, financial advice), where practitioners need to know whether a reasoning trace reflects the factors driving a decision or merely narrates a plausible-sounding process.
• Monitoring for misalignment and hidden influences, since the mediation results show that decision-relevant information can flow into predictions without appearing in the text, meaning text-only monitoring is insufficient.
• Bias detection and mitigation, where the paper points to representation-level interventions (citing Karvonen and Marks, 2025, on reducing racial and gender bias) that can find demographic directions causally affecting predictions even when CoTs never mention them.
• Debugging multi-hop QA systems, where knowing whether an answer flip is driven by a prompt artifact versus genuine reasoning changes how developers should fix prompts and evaluate models.

Industry relevance. Companies relying on CoT outputs for safety cases, red-teaming, or compliance reporting need to know that absence of a cue in the explanation is not evidence that the cue did not shape the answer. The paper's recommendation — combine CoT inspection with causal and corruption-based tests — translates directly into evaluation harness design.

Future Directions

• Distinguishing incompleteness from non-exhaustiveness. The authors state their metric does not directly separate a CoT that omits a hint due to budget limits from one that reflects a different reasoning pathway than the one that drove the prediction. They argue that separating these requires controlled settings where the active pathways and accessed information are known, and leave open whether faithful@k behaves differently under those conditions.

• Improving CoT training objectives. The paper criticizes Verbalization Finetuning (Turpin et al., 2025) because its held-out tests closely match the training data, and calls for general objectives that incentivize models to reveal implicit, real-world factors rather than toy interventions.

• Better judges for verbalization detection. The current LLM-as-judge has low precision (36%) and recall (31%); the authors note that false negatives could weaken their findings and rerun Filler Tokens and FUR on a stricter subset where hints are not even lexically verbalized, finding results unchanged.

• A broader interpretability toolkit. The paper advocates combining CoT analysis with representation-level interventions, causal mediation, and corruption-based metrics, and notes that no single metric captures faithfulness fully.

Target Audience

Researchers and practitioners in LLM interpretability, evaluation, and safety who currently use or cite hint-based CoT faithfulness results; prompt-engineering and eval-harness developers who need to know which faithfulness metrics to trust; and alignment researchers designing monitoring or auditing pipelines around reasoning traces. Readers without prior exposure to causal mediation analysis or layer-wise probing will find the conceptual arguments accessible, but the quantitative sections assume some familiarity with faithfulness-metric literature.

Authors’ abstract

Recent work, using the Biasing Features metric, labels a CoT as unfaithful if it omits a prompt-injected hint that affected the prediction. We argue this metric adopts a narrow notion of faithfulness and confuses unfaithfulness with incompleteness, the lossy compression needed to turn distributed transformer computation into a linear natural language narrative. On multi-hop reasoning tasks with instruct-tuned and reasoning models, many CoTs flagged as unfaithful by Biasing Features are judged faithful by other metrics, exceeding 50% in some models. With a new faithful@k metric, we show that larger inference-time budgets greatly increase hint verbalization (up to 90% in some settings), suggesting much apparent unfaithfulness is due to tight token limits. Using Causal Mediation Analysis, we further show that even non-verbalized hints can causally mediate prediction changes through the CoT. We therefore caution against relying solely on hint-based evaluations and advocate a broader interpretability toolkit, including causal mediation and corruption-based metrics. We do not claim all CoTs are faithful, only that the absence of hint words alone does not prove unfaithfulness.