When Agents Disagree With Themselves: Behavioral Consistency as an Uncertainty Signal for LLM Agents

AI summary

Overview

Research area: Uncertainty quantification and reliability evaluation for LLM-based tool-using agents (ReAct-style, multi-step).

Technical level: Intermediate. The paper is empirical and requires familiarity with selective classification, conformal prediction, and AUROC, but the core idea (run the agent several times, see whether it behaves the same way) is intuitive.

Scope in one sentence: The paper studies behavioral consistency across repeated runs of four frontier LLMs on HotpotQA (8,000 runs) and SWE-bench (1,000 runs), showing that cross-run disagreement is a training-free, black-box uncertainty signal usable for failure detection, selective prediction, and more reliable benchmark ranking.

What This Paper Is About

LLM agents given identical inputs do not behave identically: running the same agent on the same question produces 2.3–4.2 distinct action sequences per 10 runs. The paper asks whether this inconsistency is merely noise or a useful signal, and shows it tracks correctness. The goal is to convert cross-run agreement into a practical uncertainty estimate that requires no training, no access to model internals, no log-probabilities, and no prompt engineering.

Key Contributions

1. A filter-versus-aggregator asymmetry for agents. Majority voting over runs yields only +0–2pp in the multi-step agentic setting, versus +5–17pp reported for single-turn chain-of-thought self-consistency (Wang et al., 2023), because agent errors are systematic. Filtering on disagreement instead yields +6–14pp for three of four models, with the ceiling set by "consistent-wrong" tasks (5.5–10%).
2. Selective prediction with distribution-free validation. Cross-run agreement is used as the selection function in the Geifman–El-Yaniv selective classification framework, supported by a Hoeffding-style concentration bound (Proposition 1), reaching 87–88% accuracy at 54–62% coverage and matching a split-conformal baseline without a held-out calibration set.
3. Behavioral consistency versus verbalized confidence. In a head-to-head comparison on a common 100-question split, behavioral agreement beats verbalized self-confidence as a failure detector for Claude, GPT-5, and Llama (AUROC 0.65–0.74 vs. 0.48–0.55), and combining the two signals does not improve over behavioral consistency alone.
4. Cross-benchmark stability and ranking instability. The consistency hierarchy (Claude < GPT-5 < Gemini < Llama by coefficient of variation) is preserved exactly from HotpotQA to SWE-bench Verified, while single-run evaluations misrank models 29.3% [28.4–30.1%] of the time.

Main Findings

• Behavioral variance is universal. Across 8,000 runs of four models on 200 HotpotQA questions, Claude Sonnet 4.5 was most consistent (2.3 unique action sequences, 81.5% correct), and Llama 3.1 70B most variable (4.2 sequences, 73.7% correct). GPT-5 showed 2.4 sequences and 79.6% accuracy; Gemini 3 Pro showed 3.2 sequences and 72.2%.
• Consistency tracks correctness. Consistent tasks (≤2 unique sequences) achieve 82–87% accuracy while inconsistent tasks (≥4 sequences) achieve 41–65%. The gap is significant for all four models (Mann-Whitney U, all p < 0.001; rank-biserial r = −0.35 to −0.61) and survives Holm-Bonferroni correction (most stringent adjusted p = 3.44 × 10⁻⁵).
• Difficulty is a confound but not the whole story. After binning questions into Easy (≥0.80, n=132), Medium (0.40–0.79, n=32), and Hard (<0.40, n=36), the consistency gap remains positive in the Easy stratum for Gemini (+16.5pp), Llama (+11.2pp), and GPT-5 (+6.8pp) but is near zero for Claude (−0.5pp, a ceiling effect). Partial correlations controlling for difficulty remain significant for Gemini (r = −0.49, p < 0.001), GPT-5 (r = −0.16, p = 0.025), and Llama (r = −0.16, p = 0.028), but not Claude (r = 0.01, p = 0.88).
• The Hard-stratum gap reversal is partly a metric artifact. Of 61 consistent Hard task-model pairs, 52 (85.2%) scored incorrect under fuzzy matching, but roughly 30% of these are false negatives, such as "Pasek and Paul" versus gold "Pasek & Paul". Under token F1 ≥ 0.5, all four models show non-negative gaps (+9 to +16pp).
• Consistency detects failures at runtime. Best per-model AUROC values were 0.775 (Gemini, using >4 sequences), 0.714 (Llama, entropy >1.5), 0.688 (GPT-5), and 0.619 (Claude) — an overall range of 0.62–0.78.
• Selective prediction works, and does not need a calibration set. Answering only when k=3 runs agree gives 87.2% accuracy at 53.8% coverage (Llama), 87.8% at 61.8% (Claude), 88.1% at 55.2% (GPT-5), and 73.4% at 64.7% (Gemini). The risk-coverage curve matches a split-conformal baseline calibrated on 70% of tasks and evaluated on 30% over 500 random splits. Unanimous 5/5 agreement reaches 90.2% for GPT-5 at 48% coverage.
• Majority voting helps much less than filtering. Voting gains were +2.0pp (Llama), +0.7pp (Claude), +0.5pp (GPT-5), and +0.1pp (Gemini) at k=3. At k=10, Llama gained +4.4pp and Claude +2.0pp, while Gemini showed −0.4pp.
• Errors can be confidently wrong. Consistent-wrong tasks (5.5–10% of tasks) set the hard ceiling on filtering gains. 61% of Claude's errors are consistent-wrong versus only 25% of Llama's.
• Divergence happens early. Pooled across models, 18.8% [95% CI: 16.2–21.6%] of tasks diverge by step 2, but Llama diverges early in 50.5% [43.6–57.4%] of tasks versus 4–6% for Claude and GPT-5. Llama's early-diverging tasks reach 71.7% accuracy versus 85.8% for late-diverging tasks. Step-2 action agreement alone predicts final correctness at only AUROC 0.49–0.60.
• Path length tracks consistency. Spearman ρ between mean steps per task and unique action sequences was 0.74 (Llama), 0.54 (Gemini), 0.53 (Claude), and 0.50 (GPT-5), pooled ρ = 0.51, all p < 0.001.
• Calibration is imperfect. Expected calibration error between answer consistency and actual correctness was 0.148 (Claude), 0.153 (GPT-5), 0.172 (Gemini), and 0.090 (Llama).
• Consistency is benchmark-independent. On SWE-bench Verified (50 tasks, 5 repositories, 1,000 runs), CV was 17.7% (Claude), 30.3% (GPT-5), 46.8% (Gemini), and 68.9% (Llama) — the same ordering. Mean trajectory length ranged from 10.5 steps (GPT-5) to 81.8 (Gemini), a 7.8× ratio. Resolution rates on the original 10-task astropy subset were Claude 58%, GPT-5 32%, Llama 4%.
• Single-run benchmarks are unreliable. Across 10,000 bootstrap iterations sampling one run per question per model, 29.3% [28.4–30.1%] of evaluations produced a ranking different from the multi-run ground truth.

Methodology in Plain English

The researchers built a ReAct-style agent with three tools — Search, Retrieve, and Finish — and ran it repeatedly on the same questions. The main dataset was 200 HotpotQA validation questions in the distractor setting, each with 10 paragraphs (2 gold, 8 distractors). Four models were tested: Llama 3.1 70B, GPT-5, Claude Sonnet 4.5, and Gemini 3 Pro. Each question–model pair was run 10 times at temperature 0.7, giving 8,000 runs total. Per-run analyses and the verbalized-confidence comparison use 100-question splits with k=10 and k=5 runs respectively; a separate 100-question subset was used for a GPT-4o reference evaluation.

For each task, the team measured how many distinct action sequences appeared, how often runs agreed on the answer, how variable the step counts were, where the first divergence occurred, and whether the final answer was correct (via fuzzy string matching, with exact match and token F1 checks as robustness tests). Correctness was then compared between consistent and inconsistent tasks.

To turn consistency into a decision rule, they treated each run as an implicit ensemble member and defined an agreement score: the fraction of runs giving the plurality answer. Answering only when this score exceeds a threshold gives a selective classifier, so the work reports risk-coverage curves and compares them to split-conformal calibration. A further experiment asked each model for a post-hoc 0–100 confidence rating via one extra API call, and compared that self-rating against behavioral agreement as a failure detector. Finally, the setup was repeated on SWE-bench Verified — 50 tasks spanning 5 open-source repositories, 5 runs per model-task pair, temperature 0.5, identical bash-only scaffold — and a temperature ablation was run on 20 questions × 5 runs × 4 models × 3 temperatures. All statistical tests were two-sided and uncorrected for multiple comparisons given the exploratory nature of the study.

Why This Matters

Impact on research. The paper argues that a large fraction of published agent benchmark numbers may be unreliable: 29.3% of single-run evaluations reorder models relative to the multi-run ground truth. It also reframes self-consistency — well established for single-turn chain-of-thought — as something different in multi-step agents, where systematic early errors make majority voting weak and disagreement-filtering strong. The consistency signal is positioned as complementary to token-level semantic uncertainty, because two runs can reach the same answer through different tool sequences, a plan-level uncertainty that log-probability methods cannot see.

Real-world applications.

• Runtime monitoring of deployed agents: flagging runs where repeated executions disagree, at AUROC 0.62–0.78, without model modification.
• Selective answering or escalation: abstaining unless k=3 runs agree yields 87–88% accuracy at 54–62% coverage, at roughly $0.03–0.15 per task depending on the model.
• Model comparison and procurement: requiring multi-run evaluations rather than single-run headline numbers.
• Diagnosing agent design failures: since divergence concentrates at step 2, first-query formulation is a concrete target for reducing downstream variance.

Industry relevance. The method is black-box and training-free, works across providers, and uses only runs a system may already be performing for ensembling — no calibration split, no log-probs, no internal state access. The paper also notes a risk: consistency-based filtering could systematically abstain on underrepresented task subgroups, so it recommends pairing it with disaggregated evaluation.

Future Directions

• Extending the analysis to multimodal agents and additional domains, specifically FinQA and GAIA, which the authors name as planned extensions.
• Establishing causality: the current evidence is observational, so whether early divergence causes error or merely co-occurs with it remains open.
• Broadening cross-benchmark validation beyond the 50-task SWE-bench sample, since resolution rates are available only for the original 10-task astropy subset.
• Developing benchmarks and reporting norms for multi-run evaluation, including step-level agreement as a running test statistic for sequential monitoring.
• Testing whether improving first-query formulation reduces downstream behavioral variance, given the step-2 divergence concentration.

Target Audience

Researchers and engineers working on LLM agent reliability, uncertainty quantification, and evaluation methodology will get the most from this paper, particularly those interested in selective prediction, conformal methods, or self-consistency. Practitioners deploying multi-step agents in production will find the runtime failure-detection and abstention results directly actionable. Readers need some comfort with AUROC, calibration, and selective classification to follow the statistical sections, but the central finding — inconsistent runs are more likely to be wrong — is accessible without that background.

Authors’ abstract

Running the same LLM agent on identical inputs yields 2.3-4.2 distinct action sequences per 10 runs; this behavioral variance constitutes a training-free, black-box uncertainty signal that instantiates selective classification and distribution-free calibration for agentic systems. Across 8,000 runs of four models on 200 HotpotQA questions, consistent tasks (at most 2 unique paths) achieve 82-87% accuracy while inconsistent tasks (4 or more paths) achieve 41-65%, a gap that survives controls for task difficulty. Divergence concentrates at step 2 (50.5% of Llama tasks), and consistency metrics detect failures with AUROC 0.62-0.78. Exploiting this signal, selective prediction (answering only when k=3 runs agree) achieves 87-88% accuracy at 54-62% coverage, a 6-14pp gain over single-run baselines, and matches a split-conformal baseline without a held-out calibration set. A cross-benchmark validation on SWE-bench (50 tasks, 1,000 runs) preserves the consistency hierarchy while revealing an ~8x spread in mean trajectory length across models, and bootstrap analysis shows single-run evaluations misrank models 29.3% of the time.