Self-Evaluating LLMs for Multi-Step Tasks: Stepwise Confidence Estimation for Failure Detection

AI summary

Overview

Research area: Reliability and failure detection for large language models — specifically confidence estimation (self-evaluation) applied to multi-step reasoning tasks.

Technical level: Intermediate. The paper assumes familiarity with LLMs, fine-tuning, AUC-ROC, and binary classification metrics, but its central comparison (scoring one whole response vs. scoring each step) is conceptually simple.

Scope: The paper compares two ways of assigning confidence scores to multi-step LLM outputs — one score for the entire response versus a separate score for each reasoning step — across tool-assisted math reasoning (GSM8K) and conversational question answering (CoQA), plus a private clinical dataset.

What This Paper Is About

LLM agents that reason over multiple steps can fail at any step, and errors in early steps often propagate into later ones. Prior confidence-estimation work mostly targets single-step outputs, so it is unclear how to extend it to multi-step interactions. The authors test whether an LLM should be evaluated holistically (one confidence score for the full solution) or step by step (one score per intermediate response), and which scoring method detects failures best under each formulation.

Key Contributions

1. Extends self-evaluation to multi-step tasks. The paper formalizes failure detection for multi-step interactions where the input is a context plus a sequence of queries and the agent produces a sequence of responses, defining both response-level scoring (Equation 1) and step-level scoring (Equation 2), including a min-over-steps rule for flagging a whole trajectory.
2. A systematic comparison across two multi-step settings. The authors evaluate seven confidence estimation techniques — self-verbalized confidence, Llama-3.2-11B as an evaluator, GPT-4.1-mini as an evaluator, a regression model, a preference-based reward model (PRM), self-certainty (logit-based), and an activations-based classifier — at both granularities on GSM8K and CoQA.
3. Quantifies when step-level scoring wins. Stepwise evaluation generally outperforms holistic scoring, with up to 15% relative increase in AUC-ROC; the largest gains appear on CoQA.
4. Diagnoses correct-answer-but-flawed-reasoning cases and tests on real data. On GSM8K the agent reached the correct answer despite flawed intermediate reasoning in 60/879 test cases, and the paper reports which methods detect those cases, plus a private clinical dataset result.

Main Findings

• Step-level scoring helps most on CoQA. For CoQA, step-level scoring significantly outperforms response-level scoring across all methods. The regression model goes from AUC 0.689 (response) to 0.952 (step), a 38% relative gain with FPR@0.9 recall dropping from 0.732 to 0.169. Self-certainty jumps from 0.523 to 0.849 (62% relative gain).
• GSM8K is less consistent. Differences between granularities are smaller and trends less consistent. The regression model improves from 0.843 to 0.907 (7% relative, FPR@0.9 recall 0.441 to 0.314), but GPT-4.1-mini drops from 0.880 to 0.670 (-24%) and self-certainty drops from 0.649 to 0.395 (-40%).
• Regression model is the best performer overall. It performs best for both tasks; the best methods achieve AUC-ROC of 0.9 and recall of 0.9 with FPR below one-third.
• Preference-based reward models perform poorly. The authors suggest PRMs are better suited for ranking responses by quality than for tasks with objective correctness labels.
• Self-certainty degrades at the step level on GSM8K. The paper attributes this to tool interactions that alter the agent's responses at each step, distorting the logits; the activations-based regressor avoids this because it only relies on hidden states from the final token.
• Most techniques do better on CoQA than GSM8K, suggesting reasoning-intensive math problems are harder for evaluators than context-grounded QA. GPT-4.1-mini is the exception, showing significantly improved performance on GSM8K, which the authors attribute to its superior reasoning ability.
• Initial single-step application of self-certainty to CoQA is very weak. Applied directly, self-certainty yields AUC-ROC 0.523 and FPR@0.9 recall 0.95; a trivial extension of detecting errors after each step raises this to AUC-ROC 0.849 with FPR@0.9 recall 0.374.
• Step-level scoring is better at catching right-answer-wrong-reasoning. On GSM8K, 60/879 test cases reached the correct answer through flawed reasoning. Recall on those cases: regression 0.367 (response) vs. 0.60 (step); Llama-3.2-11B 0.133 vs. 0.40; GPT-4.1-mini 0.50 vs. 0.30; self-certainty 0.133 vs. 0.217; activations 0.167 vs. 0.267; self-verbalized 0.05 vs. 0.
• Real-world clinical data shows the same pattern. On a private dataset of conversational QA over real patient clinical notes, a regression model with step-level scores achieved AUC-ROC = 0.940 and FPR@0.9 rec = 0.152.

Methodology in Plain English

The researchers set up two controlled multi-step environments where correctness can be checked objectively. In GSM8K, the agent solves math word problems a step at a time, calling an expression evaluator tool at each step; test problems require an average of 5.1 steps. In CoQA, the agent answers a conversation's questions in sequence, where later questions depend on earlier ones; conversations average 13.5 steps.

They fine-tune Llama-3.2-11B-Instruct for two epochs to act as the agent, holding out subsets of the train and test splits for training the confidence scorers. GPT-5 is used to label whether each step and each final answer is correct, with manual review of 100 samples from each dataset confirming labeling accuracy above 96% in both settings.

They then take seven scoring methods and run each one twice: once producing a single score for the whole solution, and once producing a score per step (conditioned on the context, all previous queries, and all previous responses). Supervised scorers are trained with teacher forcing, using gold history during training, though at inference time no ground truth is available. Error detection is treated as binary classification and measured with AUC-ROC and FPR@0.9 Recall — the false positive rate at the threshold where the scorer catches at least 0.9 of the incorrect responses. When a method cannot reach 0.9 recall without flagging everything as incorrect, the paper reports FPR@0.9 recall as 1 and additionally reports the maximum achievable recall.

Why This Matters

Impact on research. The paper shows that the standard unit of analysis in confidence estimation — one query, one response — breaks down for multi-step agents, and that the choice of granularity changes measured performance substantially (in some cases by 60% relative, in others it hurts). It also shows that scoring at the step level surfaces cases where an agent gets the right answer through wrong reasoning, a failure mode that response-level scoring largely misses.

Real-world applications:

• High-stakes clinical question answering over patient notes, which the authors test directly with a private dataset (AUC-ROC 0.940, FPR@0.9 rec 0.152).
• Tool-assisted computation and multi-step math problem solving, where an evaluator can flag a bad intermediate formula before it cascades.
• Dialog systems that must decide mid-conversation whether an answer should be surfaced or escalated.
• Collaborative problem-solving and multi-hop question answering agents, where errors at any step can propagate downstream.

Industry relevance. The paper provides a practical framework for failure detection in deployed agents and shows which methods are worth the implementation cost — the regression model works best at the step level, whereas preference-based reward models, despite their popularity, underperform here. It also notes a concrete engineering tradeoff: logit-based scoring (self-certainty) is disrupted by tool calls, while an activations-based classifier that reads only the final token's hidden states is not.

Future Directions

1. Step-level training and evaluation for preference-based reward models on GSM8K. The authors leave this as future work because multiple valid reasoning paths to the same problem make generating step-level preference data infeasible — each incorrect step in an interaction would need a corrected version.
2. Confidence estimators designed specifically for multi-step reasoning. The paper states its results highlight the limits of current methods and provides a basis for developing estimators better suited to this setting.
3. Understanding why granularity effects differ by task. Step-level scoring is clearly better on CoQA but inconsistent on GSM8K, and the paper does not resolve why reasoning-intensive math is harder for evaluators.
4. Mitigating tool-call distortion of logit-based confidence. Self-certainty degrades sharply at the step level on GSM8K because tool interactions alter agent responses; the paper observes this but does not offer a fix.

Target Audience

Researchers and engineers building or deploying multi-step LLM agents who need reliable failure detection, particularly those working on confidence estimation, LLM self-evaluation, or agent reliability. It is also relevant to practitioners in regulated or high-stakes domains such as healthcare, who must decide whether to trust an agent's output before acting on it. Readers looking for a fully reproducible clinical result should note that the private dataset is not released due to conflict of interest and HIPAA compliance, and those results are therefore not reproducible.

Authors’ abstract

Reliability and failure detection of large language models (LLMs) is critical for their deployment in high-stakes, multi-step reasoning tasks. Prior work explores confidence estimation for self-evaluating LLM-scorer systems, with confidence scorers estimating the likelihood of errors in LLM responses. However, most methods focus on single-step outputs and overlook the challenges of multi-step reasoning. In this work, we extend self-evaluation techniques to multi-step tasks, testing two intuitive approaches: holistic scoring and step-by-step scoring. Using two multi-step benchmark datasets, we show that stepwise evaluation generally outperforms holistic scoring in detecting potential errors, with up to 15% relative increase in AUC-ROC. Our findings demonstrate that self-evaluating LLM systems provide meaningful confidence estimates in complex reasoning, improving their trustworthiness and providing a practical framework for failure detection.