Intention Collapse: Intention-Level Metrics for Reasoning in Language Models

AI summary

Overview

Research area: Natural language processing and large language model interpretability, specifically the internal-representation side of chain-of-thought reasoning and evaluation.

Technical level: Intermediate. The paper is readable without deep mathematics, but it assumes familiarity with transformer hidden states, Shannon entropy, PCA, logistic-regression probes, and AUROC as an evaluation metric.

Scope: The paper proposes a conceptual framework ("intention collapse") plus three cheap, model-agnostic metrics computed on a model's pre-collapse internal state, and instantiates them in a controlled 3 × 3 study across three models and three benchmarks.

What This Paper Is About

Language generation is a many-to-one mapping: a rich, high-dimensional internal state has to be compressed into a single, ordered sequence of tokens, and most of that internal content is necessarily lost. The paper names this reduction "intention collapse" and argues that the pre-collapse state is worth measuring directly, not just the final answer or the length of the reasoning trace. Its goal is to define three inexpensive metrics on that pre-collapse state and test whether they reveal structure that behavioral accuracy alone hides.

Key Contributions

1. A formal framing of intention collapse. The paper defines an intention state I as the collection of hidden representations at the final prompt position, immediately before the first output token is selected, and a collapse operator κ_θ that maps I into language. It explicitly does not claim LLMs have intentions in a conscious sense, only that both biological and artificial systems share the computational shape of rich internal dynamics followed by a sharply reduced verbal surface.

2. Three cheap, model-agnostic intention metrics. (i) Intention entropy H_int(I), the Shannon entropy of the next-token distribution at the pre-collapse moment, reported in bits with base-2 logarithms; (ii) effective dimensionality d_eff(I), the participation ratio of PCA eigenvalues over pre-collapse hidden states, computed per layer with a variance-weighted aggregation across layers; and (iii) recoverability Recov(I;Z), operationalized as probe AUROC for predicting eventual success from I.

3. A controlled 3 × 3 empirical study. Three instruction-tuned models (Mistral-7B-Instruct, Llama-3.1-8B-Instruct, Qwen-2.5-7B-Instruct) crossed with three benchmarks (GSM8K, ARC-Challenge, AQUA-RAT), under three inference regimes (Baseline, CoT, Babble), with n = 200 items per model–benchmark cell and greedy decoding at temperature T = 0.

4. A reinterpretation of reasoning techniques as interventions on I. STaR, Quiet-STaR, process reward models, test-time training, and iterative refinement or search methods (Self-Refine, MCTS, best-of-N) are recast as methods that reshape the formation of I and/or the policy that commits I to a discrete output, rather than as changes in surface verbosity.

Main Findings

• CoT is not universally beneficial. CoT raises average accuracy from 34.2% (Baseline) to 47.3%, a gain of +13.1 percentage points, driven by large gains on GSM8K across all three models. But CoT consistently degrades performance on ARC-Challenge across all three models and produces mixed outcomes on AQUA-RAT, with strong improvements for some models and substantial drops for others.

• CoT induces different entropy regimes in different model families. Comparing CoT to baseline via ΔH = H_int(CoT) − H_int(Base), Mistral operates under lower pre-collapse entropy in CoT (ΔH < 0) while LLaMA operates under higher pre-collapse entropy (ΔH > 0). For Qwen, ΔH is available only for a subset of benchmarks because baseline entropy logs are missing in some runs; the paper treats this missingness explicitly as a limitation. The paper stresses that these labels describe comparative uncertainty under CoT versus baseline and do not assume token-level temporal dynamics.

• Response format is a major axis of variation. Large within-domain gaps appear between free-response (GSM8K) and multiple-choice (AQUA-RAT) math settings, with cross-model reversals indicating that format preference is not uniform across model families. This motivates treating multiple-choice failure modes as potentially distinct from domain competence.

• Recoverability is real but can dissociate from accuracy. Linear probes trained on I reach AUROC above chance in a subset of settings. High AUROC can co-occur with degraded CoT accuracy in multiple-choice regimes — the paper cites high AUROC alongside lower CoT accuracy on ARC-Challenge for Qwen — suggesting an informative internal signal is not always reliably converted into a final discrete decision under certain response formats.

• Babble isolates verbosity but does not replicate CoT signatures. Despite producing long outputs, the babble control does not systematically reproduce the same metric–performance relationships seen under CoT, supporting the view that CoT changes more than output length.

• Probe robustness was checked with explicit safeguards. The pipeline uses item-level splits (60%/20%/20%) rather than token-level splits, z-score normalization using training statistics only, and ℓ2-regularized logistic regression with C selected from {10⁻⁴, 10⁻³, …, 10²} by validation AUROC. Shuffled-label AUROC remains near chance across cells, and the constant-score baseline yields AUROC = 0.5. Several cells show large train–test AUROC gaps, which the paper says motivates conservative interpretation.

• Cross-regime transfer suggests representational shift. Training probes on one regime and testing on another often degrades AUROC relative to within-regime AUROC, indicating that linearly accessible signals are frequently encoded along regime-specific axes.

• Not reported in this revision: option-normalized entropy and pre-collapse logit margins over multiple-choice option tokens (e.g., {A, B, C, D, E}). The paper states its checkpoint format stores aggregated entropy values but not per-option log-probabilities, and it scopes claims about H_int to the unconstrained distribution over the full vocabulary.

Methodology in Plain English

The researchers picked a single, fixed moment to look inside the model: right after the prompt has been processed but before the first output token is chosen. At that moment they save the hidden state at the last prompt position across a selected set of layers, calling this collection the intention state I. Because the same extraction point is used for every condition, differences in I can be attributed to how the prompt conditioned the model rather than to anything generated afterward.

They then compute three things from I and the surrounding forward pass. Entropy is read off the next-token distribution at that moment — a diffuse distribution means many plausible continuations. Effective dimensionality is computed by collecting hidden states across many examples, centering them, taking the sample covariance, extracting eigenvalues, and computing the participation ratio, done per layer and then combined with a variance-weighted average so that layers with different representation spaces are never concatenated. Recoverability is measured by training a linear probe on the flattened hidden states to predict whether the eventual answer was correct, with all inference regimes for a given item kept in the same split so nothing leaks across train and test. They chose AUROC because it is robust to class imbalance, and they report bootstrap 95% confidence intervals from 1000 resamples.

To make sure the results were not accidental, they ran sanity checks (a constant-score baseline and label-shuffle probes), tested whether the dimensionality estimates were stable under subsampling at N ∈ {25, 50, 100, 150, 200}, cross-checked the dimensionality trends against nearest-neighbor and maximum-likelihood intrinsic-dimension estimators, and tested whether probes trained on one regime transfer to another. The three inference regimes were a direct-answer Baseline, CoT ("think step by step"), and a Babble control that asks for long continuation unrelated to solving, which exists to separate "more tokens" from "more reasoning."

Why This Matters

Impact on research. The paper argues that benchmark accuracy is an incomplete signal because it conflates two things: what the model represents before committing, and how reliably it commits. If those can come apart — as the reported Qwen ARC-Challenge case suggests — then evaluation of reasoning methods needs to measure the pre-collapse state directly. The framework also gives a shared vocabulary for comparing techniques that are usually discussed separately (prompting, process supervision, test-time training, search), by asking how each one alters the same object I.

Real-world applications:

• Diagnosing failure modes in deployed assistants. A model that internally "knows" the right answer but fails to commit to it under a particular answer format is a different problem from a model that lacks the knowledge, and the two call for different fixes.
• Format selection for production systems. The strong free-response versus multiple-choice gaps and the cross-model reversals suggest that the choice between answer formats should be validated per model rather than assumed.
• Cost-aware inference. The paper suggests state-dependent decoding (adapting sampling temperature based on I) and early-exit strategies that use recoverability to skip unnecessary computation — relevant where reasoning traces are expensive.
• Trust and uncertainty reporting. Intention entropy and recoverability are candidate internal signals for flagging when a model's commitment is likely to be unreliable.

Industry relevance. All three metrics are described as cheap to compute, requiring no architectural modifications, and applicable to any transformer-based model, and the protocols are described as runnable on accessible open-weight models with standard libraries (Hugging Face Transformers for activation extraction, scikit-learn for PCA and probing). That lowers the barrier for teams that want internal diagnostics without retraining.

Future Directions

• Log per-option log-probabilities. The paper explicitly defers option-normalized entropy and pre-collapse logit margins over multiple-choice option tokens to future work, since the current checkpoint format only stores aggregated entropy values.
• Move from correlation to causation. The paper states that the relationship between intention metrics and downstream accuracy is correlational rather than causal, leaving causal intervention untested.
• Extend beyond the reported matrix. MATH (competition-level math) is named as a natural extension but is not part of the primary reported 3 × 3 matrix, and multimodal validation of the modality-agnostic formulation is left to future work.
• Use recoverability and entropy to drive decoding and computation. The paper proposes state-dependent decoding policies that adapt sampling temperature based on I, and early-exit strategies that use Recov(I;Z) to avoid unnecessary computation, as concrete research directions.
• Open questions the results raise: why CoT helps free-response math but consistently hurts ARC-Challenge; why ΔH flips sign between Mistral and LLaMA; and whether the observed AUROC–accuracy dissociations reflect a commitment-stage bottleneck rather than a representation-stage one.

Target Audience

Researchers and practitioners working on LLM reasoning, evaluation, and interpretability — particularly those interested in internal representations, chain-of-thought analysis, uncertainty estimation, and probing methods. It is also useful for engineers who deploy models across mixed answer formats and need to understand why the same method can help on one benchmark and hurt on another. Readers looking for a fully worked-out causal account or for standardized multiple-choice entropy diagnostics should note those are scoped out or deferred in this revision, and the provided content does not include the paper's later sections on related work, limitations, and conclusions beyond their listed outlines.

Authors’ abstract

Language generation maps a rich, high-dimensional internal state to a single token sequence. We study this many-to-one mapping through the lens of intention collapse: the projection from an internal intention space I to an external language space L. We introduce three cheap, model-agnostic metrics computed on a pre-collapse state I: (i) intention entropy Hint(I), (ii) effective dimensionality deff(I), and (iii) recoverability Recov(I), operationalized as probe AUROC for predicting eventual success. We evaluate these metrics in a 3x3 study across models (Mistral-7B, LLaMA-3.1-8B, Qwen-2.5-7B) and benchmarks (GSM8K, ARC-Challenge, AQUA-RAT), comparing baseline, chain-of-thought (CoT), and a babble control (n=200 items per cell). CoT increases average accuracy from 34.2% to 47.3% (+13.1 pp), driven by large gains on GSM8K but consistent degradations on ARC-Challenge. Across models, CoT induces distinct entropy regimes relative to baseline, dH = Hint(CoT) - Hint(Base): Mistral shows dH &lt; 0 (lower-entropy CoT), whereas LLaMA shows dH &gt; 0 (higher-entropy CoT), highlighting heterogeneity in CoT-induced internal uncertainty. Finally, probe AUROC is significantly above chance in a subset of settings and can dissociate from behavioral accuracy (e.g., high AUROC alongside lower CoT accuracy on ARC-Challenge for Qwen), suggesting that informative internal signal is not always reliably converted into a final discrete decision under constrained response formats.