LLMs Encode Their Failures: Predicting Success from Pre-Generation Activations

AI summary

Overview

Research area: Natural Language Processing / LLM interpretability and inference efficiency — specifically, extracting signals of a model's own likely success from its internal activations before it generates any output, and using those signals to route queries between models.

Technical level: Intermediate. The methods rest on linear probes and standard metrics (Spearman correlation, AUROC), but the paper explains the setup clearly enough for readers comfortable with basic machine learning.

Scope: An empirical study across mathematics and coding benchmarks showing that LLMs encode a model-specific notion of difficulty in pre-generation activations, that this notion diverges from human difficulty under extended reasoning, and that probes extracting it can cut inference cost substantially.

What This Paper Is About

Running an LLM with extended reasoning on every question is expensive, but it is hard to know in advance which inputs actually need that extra compute. The authors ask whether a model's probability of answering a given question correctly is already recoverable from its internal representations before generation begins, and whether that signal can drive cheaper, smarter model allocation. They find that it can, using simple linear probes trained on pre-generation activations.

Key Contributions

1. Human and model difficulty are encoded differently. Using E2H-AMC (the AMC subset of Easy2Hard-Bench), which supplies both human IRT difficulty labels and model accuracy on identical questions, the authors show linear probes can extract both signals from pre-generation activations (Spearman ρ = 0.83–0.87 for human difficulty, 0.40–0.64 for model difficulty). These are distinct pieces of information: model-derived difficulty is more predictive of actual model performance, and the two representations diverge as reasoning budgets increase.

2. Probes reliably predict success across decoding settings and reasoning modes. Binary classification of success under fixed decoding policies (greedy, Maj@K) reaches AUROC > 0.7 for several models and stays stable across sampling temperatures and majority-voting thresholds. Reliability degrades with extended test-time compute, but this is partially recoverable with non-linear MLP probes.

3. Probe-guided routing achieves large cost savings with minimal accuracy loss. Simple threshold-based and utility-maximizing routing policies match the strongest single model's performance at up to 70% lower inference cost on MATH, with similar gains on AIME and GSM8K. In some configurations the router exceeds the best baseline and approaches oracle-level accuracy, indicating that reliable difficulty estimates — rather than routing sophistication — are the key to effective model allocation.

4. A mechanistic account of why difficulty diverges from human judgment. Analysis of chain-of-thought length shows that longer reasoning traces track human IRT difficulty rather than the model's own likelihood of failure, helping explain why probes degrade exactly when reasoning is most effective.

Main Findings

• Human difficulty is more linearly decodable than model difficulty. On E2H-AMC, linear probes reach Spearman ρ = 0.83–0.87 for human IRT difficulty across Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, and GPT-OSS-20B (low/medium/high), versus ρ = 0.40–0.64 for the expected success rate ŝ_MC. Both beat TF-IDF (0.72–0.74 for human difficulty; 0.25–0.47 for model difficulty) and a question-length baseline (0.15 for human difficulty; 0.19–0.30 for model difficulty).

• Model-success signals get harder to read as reasoning increases. For GPT-OSS-20B, the expected-success-rate probe drops from ρ = 0.58 (low reasoning) to ρ = 0.40 (high reasoning), despite higher task success. Non-linear MLP probes do not recover this loss and degrade faster than linear probes.

• Binary success under fixed decoding is much more predictable than success rate. Probes for Maj@5 or greedy success reach AUROC > 0.7 in most settings, with several above 0.8. Greedy decoding generally yields higher probe AUROC than Maj@5 for the same model (Qwen2.5-Math-1.5B: 0.84 vs 0.76), which the authors attribute to less noise in the deterministic prediction target.

• Reasoning budgets degrade probe quality, partially recoverable non-linearly. For GPT-OSS-20B under fixed Maj@5 decoding, AUROC falls from 0.78 (low) to 0.64 (high) while task success rises from 0.866 to 0.920. At high reasoning, an MLP probe reaches 0.76 versus the linear probe's 0.64.

• Human and model difficulty diverge with extended reasoning. The Spearman correlation between probe-predicted model difficulty and human IRT difficulty drops from roughly 0.65 to roughly 0.45 as GPT-OSS-20B's reasoning level rises, yet the probe still outperforms human difficulty labels at predicting Maj@5 failures in every reasoning mode.

• Chain-of-thought length tracks human difficulty, not model uncertainty. On E2H-AMC, output length increases with human IRT difficulty but decreases with both empirical and probe-predicted success, and this pattern strengthens at higher reasoning budgets. The model spends more tokens on problems humans find hard even when those problems are well within its own competence.

• Probe-guided routing delivers benchmark-dependent savings. On MATH, the router matches GPT-OSS-20B-high's 92% accuracy at a 70% cost reduction. On AIME 2025 (40%–93% model accuracy spread), it matches the strongest model at a 37% cost reduction ($1.15 vs $1.75). On GSM8K, where performance saturates (85%–95%), it selects Math-7B (94.5% at $0.34) over GPT-OSS-20B-high (94.4% at $2.4).

• Code probes are strong where models are weak. On LiveCodeBench with Pass@5 targets, probes reach AUROC 0.91 for Qwen2.5-Coder-3B and 0.90 for Qwen2.5-Coder-7B, whose Pass@5 task success rates are only 0.14 and 0.15 respectively.

• Routing gains are bounded by probe reliability. The routers approach oracle-utility performance when probes discriminate well and show substantial gaps when probe quality degrades, indicating that success-estimate reliability, not routing policy design, is the limiting factor.

Methodology in Plain English

The authors take a pool of models — GPT-OSS-20B (low/medium/high reasoning), DeepSeek-R1-Distill-Qwen-7B, Qwen2.5-Math (1.5B, 7B), Qwen2.5-Coder (3B, 7B), and Qwen2.5-1.5B — and run them repeatedly on the same questions from MATH, GSM8K, AIME (1983–2024), AIME-2025, E2H-AMC, and LiveCodeBench. For each question they record whether the model got it right, building labels for several definitions of "difficulty": human IRT difficulty scores (available in E2H-AMC, calibrated from secondary-school AMC examinee performance), a continuous expected success rate estimated by Monte Carlo rollouts, and binary outcomes under greedy decoding, Maj@K, and Pass@K. They use K=5 rollouts for the larger GPT-OSS-20B and K=50 for the smaller Qwen models.

They then freeze the model, feed in each prompt, and read the residual-stream activations at the positions immediately following the instruction (identified via each model's chat template). On these activation vectors — swept across layers, token positions, and L2 regularization strengths with 5-fold cross-validation — they train simple linear probes: ridge regression for continuous targets, logistic regression for binary targets. They compare against a question-length baseline, a TF-IDF linear model, and a two-layer MLP probe (hidden size 256, ReLU) to test whether any signal is merely non-linear rather than absent. Probes are selected on a validation split and evaluated once on held-out test data.

For routing, they train one probe per model in a five-model pool and select the model maximizing predicted success minus a cost penalty, sweeping the trade-off parameter to trace a performance-cost frontier. Costs come from Fireworks AI inference pricing. Baselines are an IRT-based router using learned latent item and model traits, random routing, and an oracle with ground-truth correctness.

Why This Matters

This work reframes difficulty as something a model internally represents about itself, not just a property of the question. It shows that the internal signal is accessible, actionable, and systematically different from human intuition — and that the difference widens precisely as models get better at reasoning.

Impact on research:

• Provides the first direct comparison of human and model difficulty representations on identical questions, using E2H-AMC, where both IRT labels and model performance exist.
• Gives the first systematic account of how test-time scaling (both majority voting and extended reasoning) affects the linear accessibility of difficulty information, not just accuracy.
• Challenges the assumption that chain-of-thought length signals difficulty or uncertainty, complementing concurrent findings that longer traces are not reliable indicators of correctness.

Real-world applications:

• Cost-aware serving: routing easy queries to cheap models and hard ones to expensive reasoning models, cutting inference bills by 17–70% in reported configurations without losing accuracy.
• Curriculum and data selection: identifying which problems are genuinely hard for a given model rather than relying on human difficulty labels that may mischaracterize model challenges.
• Evaluation design: building benchmarks that separate human-calibrated difficulty from model-relative difficulty, since the two diverge with capability.
• Cascade and abstention systems: deciding when to attempt an answer, escalate to a stronger model, or abstain, using a probe rather than a full generation.

Industry relevance: Inference cost is a dominant operational expense for LLM deployments. A probe that reads activations before generation adds negligible overhead compared to multi-rollout confidence estimation or extra API calls, and requires no separate embedding model at routing time — unlike IRT-based routers that depend on learned latent representations. The finding that routing effectiveness is limited by probe reliability rather than policy sophistication gives engineering teams a clear target for investment.

Future Directions

• Probe beyond pre-generation. The authors restrict probes to a single post-instruction position and do not probe during generation; mid-generation probing might recover signal lost under extended reasoning.
• Cross-domain transfer. Math-to-code transfer is explicitly not tested. Whether a probe trained on math problems predicts coding success is an open question, and would determine how broadly a single router can be deployed.
• Adaptive rather than fixed k. The routing policies use fixed-k majority voting rather than learned or adaptive selection of k, leaving the choice of how many samples to draw as an unoptimized degree of freedom.
• Mitigating representation drift. The degradation of linear probe quality under extended reasoning (AUROC 0.78 → 0.64 for GPT-OSS-20B) is only partially addressed by MLP probes, and MLPs degrade more severely for continuous success-rate regression. Whether the underlying signal becomes genuinely less accessible, or merely non-linear, remains unresolved.

Target Audience

This paper is most useful to LLM interpretability researchers, inference-efficiency and serving engineers, and practitioners building model-routing or cascade systems. It also speaks to evaluation and benchmark designers interested in how human-calibrated difficulty diverges from model-relative difficulty, and to researchers studying test-time compute scaling who want to understand what extended reasoning does to a model's internal representations. Readers should be comfortable with linear probing, AUROC, and Spearman correlation, but no specialized background beyond standard machine learning is assumed.

Authors’ abstract

Running LLMs with extended reasoning on every problem is expensive, but determining which inputs actually require additional compute remains challenging. We investigate whether their own likelihood of success is recoverable from their internal representations before generation, and if this signal can guide more efficient inference. We train linear probes on pre-generation activations to predict policy-specific success on math and coding tasks, substantially outperforming surface features such as question length and TF-IDF. Using E2H-AMC, which provides both human and model performance on identical problems, we show that models encode a model-specific notion of difficulty that is distinct from human difficulty, and that this distinction increases with extended reasoning. Leveraging these probes, we demonstrate that routing queries across a pool of models can exceed the best-performing model whilst reducing inference cost by up to 70\% on MATH, showing that internal representations enable practical efficiency gains even when they diverge from human intuitions about difficulty. Our code is available at: https://github.com/KabakaWilliam/llms_know_difficulty