Modeling Hierarchical Thinking in Large Reasoning Models

AI summary

Overview

Research area: Interpretability and inference-time control of Large Reasoning Models (LRMs), combining finite-state modeling of Chain-of-Thought (CoT) with activation steering and planning-based control.

Technical level: Advanced. The paper assumes familiarity with transformer hidden states, activation steering, Q-value iteration, and Chain-of-Thought reasoning.

Scope: The paper abstracts CoT reasoning into a six-state Finite State Machine (FSM), shows that success- and failure-associated transitions can be quantified in a Transition Advantage Matrix, and uses that matrix to drive training-free, sparse activation steering during inference across four benchmarks and three open reasoning models.

What This Paper Is About

Large Reasoning Models generate long chain-of-thought sequences, but it is unclear how a model moves between cognitive phases, and therefore unclear why a given reasoning trajectory succeeds or fails. Existing work targets single behaviors or suppresses "unproductive" patterns, but lacks a global view of how reasoning modes evolve, and cannot answer what the best next cognitive move is given where the model currently is. This paper models CoT as a trajectory through six abstract reasoning states, measures which state transitions distinguish correct from incorrect solutions, and uses that structure to steer generation toward outcome-aligned transitions.

Key Contributions

1. FSM abstraction of reasoning. A formal state-space taxonomy decomposing Chain-of-Thought into a discrete trajectory of six functional cognitive states: initialization, deduction, augmentation strategy, uncertainty estimation, backtracking, and final conclusion.

2. Transition Advantage Matrix. A quantitative metric, defined as the difference between the outcome-conditioned transition matrix for correct solutions and the one for incorrect solutions, identifying which cognitive shifts are statistically associated with correct versus incorrect outcomes.

3. Planning-aware steering policy. A training-free inference-time control framework that treats reasoning as a planning problem; via Q-value iteration it estimates long-horizon utilities and uses them to gate and guide activation steering at sentence boundaries.

4. Efficiency and performance. FSM-guided steering improves performance on complex reasoning tasks (reported as +13% accuracy on AIME25 for gpt-oss-20b under low reasoning effort) while substantially reducing the number of interventions relative to greedy and weighted heuristic baselines.

Main Findings

• Accuracy gains concentrate on hard tasks: For GPT-L (gpt-oss-20b, low effort) on AIME25, accuracy rises from 43.30% (default) to 56.67% under both weighted and Q-Value steering. On MATH-500, GPT-L peaks at 83.20% with Q-Value steering versus 81.20% with greedy. On GPQA Diamond, GPT-M improves from 64.14% to 67.17% with Q-Value steering.

• Sparse interventions: Q-Value steering matches or outperforms other methods with the fewest interventions. On MATH-500, GPT-L reaches peak performance with 0.476 interventions per sample versus 12.17 under greedy steering, described as roughly a 25x reduction.

• Short-sighted steering can hurt: Greedy steering sometimes lowers accuracy — QWEN on AIME25 drops from 83.3% to 76.67%, and GPT-M on MATH-500 drops from 86.40% to 85.40% — indicating that pushing the model toward local high-reward states can lead into suboptimal paths that are hard to correct later.

• Token efficiency: Q-Value steering often reaches equal or higher accuracy with comparable or fewer tokens. On GSM8K with GPT-L, average CoT length falls from 84.05 to 73.34 tokens. Weighted steering frequently increases token usage on AIME25 (GPT-L: 1852 vs. 1725 default).

• Cross-model transfer is competitive: Applying QWEN's advantage matrix to GPT-L on MATH-500 yields 82.80% accuracy with 5.37 average interventions, versus 83.20% and 0.48 for the model-specific setup. Under greedy steering, cross-model transfer marginally exceeds the model-specific baseline (81.80% vs. 81.20%).

• Prompt-based steering works, but less well: A prompt-based alternative that uses the advantage matrix to explicitly specify desired and undesired behaviors achieves meaningful gains but consistently underperforms activation-based steering; details are deferred to the appendix.

• Annotation reliability: Manual review of 10% of AIME25 and GPQA Diamond annotations by two independent raters produced a Cohen's Kappa of 0.89. State classifiers achieved over 90% test accuracy across all settings.

Methodology in Plain English

The researchers break each chain of thought into sentences rather than tokens, then use GPT-4o-mini (temperature near 1e-19) to label every sentence with one of six reasoning states. This turns a text into a sequence of state labels, like a trail of where the model's thinking went.

From many such labeled trails, they count how often each state follows each other state, producing transition probabilities. They build two versions: one from problems the model got right, one from problems it got wrong. Subtracting the wrong-problem matrix from the right-problem matrix gives the Transition Advantage Matrix, where a positive entry means a transition shows up more in successful reasoning and a negative entry means it shows up more in failures.

To control reasoning, they first extract a "steering direction" for each state transition by comparing the average hidden activation at sentence boundaries where that transition occurred against the average for all other transitions. They drop the part of that direction parallel to the current hidden state and inject only the perpendicular part, nudging the model without overwriting its content. Classification of the current and next state is done by a small encoder trained with triplet loss projecting hidden states into a 512-dimensional normalized space, plus two lightweight classifiers trained with an 80-20 split.

Steering is only applied at sentence boundaries, detected by punctuation. Two control policies are compared with a weighted blend: greedy selection picks the single highest-advantage next state, while Q-Value steering runs 100 iterations of Q-value iteration with discount factor 0.9 to estimate long-horizon utility, clipping rewards to a range of 0.2 to 0.3. At inference, the model skips intervention when it is not stuck and its next-state confidence is at least 0.90; otherwise it steers only when the gap between the optimal and predicted transition exceeds a threshold of 0.06, with steering strength scaled by the gap times confidence.

Why This Matters

• Impact on research: The work connects interpretability to actionable control, moving from identifying steerable behaviors to deciding when and where to intervene along a trajectory. It offers a compact, transferable representation of reasoning structure and shows that the advantage matrix from one model can guide another.

• Real-world applications:

  • Efficient inference in production, where a small number of targeted interventions can improve accuracy without large compute increases.
  • Reliability engineering for reasoning systems, using the advantage matrix to flag failure-prone transition patterns.
  • Debugging and diagnosis of model outputs, since failed trajectories can be described in terms of state transitions rather than raw token streams.
  • Test-time control in settings where model weights are frozen, since no weight updates are required.

• Industry relevance: The claimed reduction in interventions (roughly 25 times fewer than greedy on MATH-500 for GPT-L) and the low token overhead of activation steering translate directly into serving cost and latency considerations for deployed reasoning models.

Future Directions

• Richer state models: The authors note that the FSM as formulated is memoryless, and that more complex models admitting model state or memory may capture reasoning dynamics more accurately; they leave this to future research.

• Domain-specific taxonomies: The six-state taxonomy was found sufficient for complex mathematics and scientific knowledge QA, but the authors state the taxonomy may vary by domain, leaving other domains to be tested.

• Stronger transfer and prompt-based control: Cross-model activation steering is only partially explored (one model pair on one dataset), and prompt-based steering using the advantage matrix is described as a promising complementary direction.

• Robustness of planning: The authors acknowledge that Q is derived from observed data rather than an absolute ground truth and is not guaranteed to be globally optimal, and that sentence-granularity inference-time methods cannot reliably distinguish sentence-ending punctuation from decimals or equation symbols.

Target Audience

Researchers and engineers working on reasoning-model interpretability, inference-time control, and activation steering will benefit most. It suits readers already comfortable with transformer internals and reinforcement-learning style value iteration, and is also useful for practitioners seeking training-free accuracy improvements for open reasoning models on math and science benchmarks.

Authors’ abstract

Large Reasoning Models (LRMs) solve complex tasks by generating long Chain-of-Thought (CoT) sequences; however, the emergent dynamics governing reasoning trajectories are not well understood and can lead to inconsistencies and reasoning pathologies. In this work, we propose to approximate LRM's emerging hierarchical reasoning dynamics as a trajectory within a Finite State Machine (FSM) transitioning among six abstract cognitive states. We demonstrate that these states and transitions can be captured in the latent state of the model. We believe that this representation can have different applications in the interpretability and optimization of LRM models. For example, by analyzing the topology of these transitions, we identify statistical shifts in reasoning strategies that help identify effective reasoning chains from those that fail. To illustrate these potential advantages, we propose Q-Value guided steering, a training-free inference-time control method that treats reasoning as a planning problem. We estimate the long-horizon utility of state transitions and apply sparse, orthogonal activation steering at sentence boundaries to align the CoT generation with optimal reasoning policies. Experiments across four benchmarks (AIME25, MATH-500, GSM8k, and GPQA Diamond) using three state-of-the-art open reasoning models demonstrate that Q-Value steering policy achieves significant performance gains with "surgical" efficiency, often requiring 25 times fewer interventions than greedy and weighted baselines, which suggests that reasoning can be effectively controlled by guiding high-level cognitive dynamics rather than micro-managing token generation. Code is available at: https://github.com/shahariar-shibli/CoT-FSM.