Legibility is Not Interpretability: Comparing Judged and Actual Importance in Chain-Of-Thought Reasoning

AI summary

Overview

Research area: Natural language processing, specifically mechanistic and behavioral interpretability of chain-of-thought (CoT) reasoning in large language models, with connections to reinforcement learning (RL) credit assignment and CoT faithfulness evaluation.

Technical level: Intermediate. The paper is readable without deep RL background, but it assumes familiarity with concepts such as value functions, advantage, Monte Carlo estimation, and process reward models.

Scope: The paper proposes measuring the importance of a CoT step as its RL-style advantage (the change in expected reward from including that step), validated via Monte Carlo rollouts and changepoint detection, and then tests whether that importance can be recovered from the text of the reasoning trace alone by LLM judges and fine-tuned critics.

What This Paper Is About

Chain-of-thought traces look legible, so a growing body of work uses LLM judges, critics, and process reward models to diagnose errors, evaluate faithfulness, and supervise reasoning step by step. These practices assume the text of a step carries information about its functional role in producing the final answer. The paper asks two questions: what is an appropriate measure of a reasoning step's importance, and is that measure decodable from the text alone? The authors answer the first by defining importance as a step's advantage under the model's own policy, and the second by showing that capable judges and fine-tuned critics can recover only part of that signal, and much less for responses that end in a correct answer.

Key Contributions

1. Operationalizing step importance as advantage. The paper defines the importance of a reasoning step as its advantage in the RL sense: the change in expected reward from committing to that step versus continuing from the preceding prefix, estimated by Monte Carlo rollouts from each prefix. Importance is measured under two rewards: correctness-based (final answer is correct) and self-advantage (final answer matches the original trace's answer).

2. A changepoint-based labeling method for consequential steps. Rather than testing each step independently, the authors model the per-prefix value trajectory as piecewise constant, segment it with the Pruned Exact Linear Time (PELT) changepoint algorithm under an exact binomial cost, and label a step consequential only if a Beta-posterior test says its jump exceeds an effect size of δ = 0.1 with probability at least 0.95. A response is uninformative if all its steps are uninformative.

3. Using advantage as an interpretive tool. The measure is applied to characterize reasoning behavior across step types, model scales, thinking modes, dataset difficulty, and cue-based faithfulness settings, connecting the analysis of reasoning to the credit assignment problem in RL and complementing perturbation-based faithfulness tests.

4. An evaluation of whether advantage is decodable from text. Out-of-the-box judge LLMs and fine-tuned step-level critics are tested against the advantage-derived labels on in-distribution and out-of-distribution math data, with split-half noise ceilings to contextualize every metric, and results stratified by whether the response's final answer is correct or incorrect.

Main Findings

• Text importance is only partially recoverable. Sufficiently capable out-of-the-box judges outperform a prevalence baseline but fall well short of a noise ceiling. The best judge (Qwen3.6-27B) remains 9× below the in-distribution noise ceiling and 6× below the out-of-distribution noise ceiling.

• Fine-tuning helps most for incorrect responses. Fine-tuned critics reach PR-AUC 0.28–0.30 (in-distribution) and 0.18–0.32 (out-of-distribution) on incorrect responses, which is 10–15× chance and roughly half the conservative ceiling of approximately 0.6.

• Correct responses remain hard. On correct responses, critics attain only 0.065–0.10 PR-AUC, which is 3.5–5× chance but only 10–20% of the corresponding ceiling of 0.51–0.64. The authors note that identifying consequential steps in correct responses is arguably the more interesting case, since these correspond to true "aha!" moments.

• High precision only at tiny budgets, and only for incorrect responses. Precision@0.5% on incorrect responses is 0.55–0.60 (in-distribution) and 0.55–0.72 (out-of-distribution), matching conservative ceilings of 0.61–0.62, but precision drops to roughly half the noise ceiling at a 2% inspection budget. On correct responses, precision@0.5% reaches only 0.08–0.16 (in-distribution) and 0.18–0.29 (out-of-distribution) against ceilings of 0.72 and 0.56 — a 5–10× improvement over chance precision of ≈0.02, but still only 10–50% of the noise ceilings.

• Critic scaling is largely flat. Unlike judges, which improve with scale, critic performance is roughly the same at 1.7B as at Qwen3.6-27B on in-distribution data. Judging and fine-tuning on correctness-based advantage performs even worse across the board.

• Consequential steps are rare. They make up 1.8% of steps in-distribution and 2.3% out-of-distribution for non-thinking Qwen3-1.7B, and are even rarer with thinking mode on.

• Step types differ by outcome. For non-thinking Qwen3-1.7B, uncertainty management and self-checking steps are most often consequential in correct responses, while final answer emission leads in incorrect responses — suggesting the non-thinking model often invents an incorrect answer at the very end. For the thinking model, active computation and uncertainty management are most often consequential in correct responses.

• Consequential steps are more common with thinking mode off than on.

• Thinking and scale help before reasoning starts. Performance gains are driven mainly by more responses whose value is high from the very first step: "high throughout" rises from 24% to 61% with thinking and from 25% to 39% across model sizes, with mass absorbed almost entirely from "never high." Gains come from problems effectively solved before the response begins, not from recovering mid-trajectory.

• Models rarely start committed to a wrong answer. Only 1–3% of non-thinking responses are locked into the wrong answer from the start, versus 12.4% of thinking responses. The thinking model reaches incorrect answers via gradual climb over twice as often as by sudden climb (59% vs. 24%), while non-thinking models reach incorrect answers more often by sudden climb (48–56%) than gradual climb (39–46%).

• Easier datasets show fewer consequential steps. GSM8K and MATH500 have a lower proportion of self-advantage steps, while the harder AIME datasets almost always have at least one consequential reasoning step.

• Advantage complements cue-based faithfulness tests. On 100 Scruples examples, without a cue the model already agrees with the cue-suggested answer 37% of the time, and with the cue 95% of the time. 58% of base responses contain at least one consequential step, versus 15% with the cue, suggesting the cue often effectively determinizes the reasoning process.

Methodology in Plain English

The authors treat a reasoning trace as a sequence of steps and the language model as a policy. The value of a prefix is the probability that completions from that prefix reach the target answer; the Q-value after committing to a step is the same probability from the prefix plus that step. Advantage is the difference between the two, so a positive advantage means the step improves the odds of reaching the target, and a negative advantage means it hurts.

Because these quantities are expectations over stochastic generation, they are estimated by Monte Carlo sampling: 50 completions are drawn at each reasoning step to estimate the value of each prefix. To decide which steps are genuinely consequential rather than noisy, the authors use changepoint detection rather than testing each step in isolation. The per-prefix value estimates are modeled as a piecewise-constant time series and segmented with PELT under an exact binomial cost. The penalty term is calibrated per model, thinking mode, and dataset using 1,000 simulated flat null series, so that flat trajectories trigger a false boundary at most 2% of the time. A detected boundary is kept only if a Beta-posterior comparison of the segments before and after shows the jump exceeds δ = 0.1 with probability at least 0.95.

Experiments use math reasoning problems: 30 problems from each of AIME 24, AIME 25, AIME 26, AMC 23, MATH500, and GSM8K, with 10 responses per prompt from Qwen3-1.7B, Qwen3-4B, and Qwen3-8B with thinking mode off, totaling 1,800 responses across 180 questions per model. The same process is repeated for Qwen3-1.7B with thinking mode on for AIME 24, AIME 25, and GSM8K, filtering the tail of long responses (over 600 steps) and leaving 81% of responses.

To test decodability, judges (Qwen3-1.7B, Qwen3-8B, Qwen3-32B, Qwen3.6-27B, thinking mode on) are prompted to rate a step's advantage in [−1, 1] in a single call. Critics add regression heads on top of the same four backbones and prompt templates and are trained on 80% of five math reasoning datasets to predict value, from which advantage is derived. Evaluation uses the held-out 20% split for in-distribution data and AMC 23 for out-of-distribution data, with PR-AUC and precision@k% because consequential steps are so rare. Since gold labels are themselves Monte Carlo estimates, an oracle could not score 1.0, so every metric is contextualized with split-half noise ceilings from half of each response's rollouts against labels from the other half (conservative) or from the full set (optimistic).

Why This Matters

Impact on research. The paper adds to a body of CoT faithfulness work cautioning against reading the legibility of reasoning traces as interpretability. It gives the field a principled, operationally defined measure of step importance connected to RL credit assignment, and it demonstrates that the text of a step encodes much less about its functional role than judge-based and critic-based methods implicitly assume — an asymmetry that is especially pronounced for correct responses. The findings have direct implications for process reward models and step-level supervision that depend on step-level text comprehension.

Real-world applications:

• Process reward models and step-level supervision. Systems that reward intermediate reasoning steps should not assume the text of a step reliably indicates its contribution; the paper measures how much signal is actually available.
• LLM-as-a-judge evaluation of reasoning. Error diagnosis and faithfulness evaluation pipelines that rely on judges reading CoT steps should account for the measured gap between judged and actual importance.
• Generative critics and error detection. The strong performance on incorrect responses and weak performance on correct responses suggest critics are better suited to catching failures than to identifying the steps that drive success.
• Benchmark and evaluation design. The finding that consequential steps are rarer on easier, better-attested datasets, and that gains from scale and thinking come from high initial value rather than mid-trace discovery, informs how reasoning capability is benchmarked.

Industry relevance. The paper was primarily conducted during an internship at Cohere, with Cohere providing compute and support, and it targets tools that companies use to train, evaluate, and improve reasoning models. Its warning is practical: if a critic or reward model is trained on step text, it may learn easy proxies, such as the final answer emission step in incorrect responses, that inflate measured performance without capturing the reasoning that produced a correct answer.

Future Directions

• How to convey step influence without over-reading text. The conclusion states an unresolved tension: how to communicate that certain reasoning steps influence answers without implying their text can be read as meaningful.
• Is the correct/incorrect asymmetry partly spurious? The authors suggest incorrect responses often contain an easy proxy step — the final answer emission — that can inflate critic performance, and ask how much of the asymmetry reflects genuine difficulty versus this artifact.
• Whether more capable critics or more data can close the gap. Critic scaling is largely flat while judges improve with scale, leaving open whether the limitation is in the probes or in the text itself, especially given that the probes are themselves highly expressive LLMs.
• Statistical power and cost of ground truth. A single two-proportion comparison at 50 rollouts has a worst-case minimum detectable effect of approximately 0.28 at 80% power, and resolving a jump of 0.1 that way would require about 392 rollouts per prefix, versus 25-step segments pooling 1,250 rollouts — a design trade-off that constrains how finely importance can be measured.

Target Audience

This paper is most useful to researchers working on chain-of-thought faithfulness and interpretability, practitioners building process reward models, LLM judges, or step-level critics for reasoning models, and evaluators who design benchmarks or supervision signals for mathematical reasoning. It is also relevant to RL researchers interested in how credit assignment concepts apply to language model reasoning traces. Readers should be comfortable with basic reinforcement learning vocabulary such as value, Q-value, and advantage, and with standard classification metrics such as PR-AUC and precision@k.

Authors’ abstract

Reasoning traces from chain-of-thought models appear to offer a legible window into how a model arrives at its answer. A growing body of work treats them as such, using LLM judges to diagnose errors, evaluate faithfulness, and provide step-level supervision via process reward models and generative critics. These practices rely on the text of a reasoning step carrying information about its functional role. But does the text actually encode information about which reasoning steps matter? We operationalize the importance of a reasoning step as its advantage: the change in expected reward, e.g., producing the correct final answer, from including that step, estimated via Monte Carlo rollouts. Basing ground truth on these estimates, we evaluate whether LLM judges can identify high-advantage steps and find that sufficiently capable LLMs can outperform a prevalence baseline but fall well short of a noise ceiling. Fine-tuning a model as a step-level critic yields strong improvement for incorrect responses but remains distant from ceiling for correct responses, suggesting that step importance is only partially recoverable from the text of the reasoning trace. Our findings contribute to a growing body of chain-of-thought faithfulness work that cautions against treating the legibility of reasoning traces as interpretability, especially with implications for process reward modeling.