Accurate but Not Humble: Evaluating Epistemic Humility in LLM Agents under Knowledge Conflict

AI summary

Overview

Research area: Evaluation of LLM-based agentic systems, specifically behavioral evaluation of how agents handle knowledge conflict (situations where retrieved evidence contradicts the backbone model's parametric beliefs or where two contextual sources disagree).

Technical level: Intermediate. The core concepts are accessible, but the paper defines formal trajectory-level metrics, uses matched conflict/control designs, bootstrap confidence intervals, and multi-judge validation.

Scope: The paper proposes and applies a trajectory-level framework for measuring epistemic humility in agents along three behavioral dimensions, Identify, Solve, and Escalate (ISE), across controlled and naturally occurring knowledge conflicts.

What This Paper Is About

Existing evaluations of LLM agents grade almost entirely on outcome: whether the final answer is correct. That leaves an important behavior unmeasured: whether an agent notices that its retrieved evidence contradicts its own prior belief, tries to resolve the contradiction with tools, and tells the user when the contradiction is still unresolved. The paper builds a way to measure this behavior, which it calls epistemic humility, and shows that high benchmark accuracy does not imply it.

Key Contributions

1. A definition and measurement model for epistemic humility in agents. The authors treat epistemic humility as an unobservable theoretical construct and decompose it into three observable, trajectory-level behaviors: Identify the gap, Solve it through bounded tool use, and Escalate residual uncertainty to the user. Each dimension has its own formal metric (Identify is scored as F1 over conflict versus no-conflict instances using only intermediate turns; Solve is a bounded-resolution rate over trajectories where a gap was flagged; Escalate is a rate over conflict-split instances that were answered incorrectly).

2. A conflict-identification pipeline producing matched conflict and no-conflict task pairs in two settings. Controlled conflict uses factual questions from ConflictQA and WikiContradict, with conflicts supplied in the task context between two passages or between context and parametric knowledge; these tasks are answerable without tools, but agents keep their full tool set and are not restricted to a single step. Naturally occurring conflict extends the evaluation to GAIA, MoNaCo, and BrowseComp, where conflicts may arise mid-execution from tool results or the model's own outputs.

3. An evaluation of four agent classes / five harness–backbone configurations (Nemotron-ToolOrchestra with Nemotron-Orchestrator-8B; Claude Code with Claude Sonnet 4.6; OpenHands with GPT-5, listed as GPT-5.2 in Table 2; and Qwen-Agent with Qwen3.5-9B and Qwen3.5-27B), scored by three judges from different model families (GPT-5, Claude Sonnet 5, GLM-5.2) and validated against human annotators.

4. An empirical result linking task accuracy to lower epistemic humility, plus a minimal system-prompt intervention and the release of every agent trajectory and the per-turn ISE judgments used for scoring (data and code at https://github.com/KaiserWhoLearns/EpistemicHumilityLLMAgents).

Main Findings

• Knowledge conflict degrades task performance. On the controlled-conflict benchmark (TMD), where the provided context contains both evidence supporting and contradicting the agent's parametric belief, all configurations with reported results lose more than 30 percentage points of accuracy relative to their no-conflict controls (for example, Nemotron-Orch-8B drops from 39.0 to 3.0, OpenHands from 58.4 to 3.2, Qwen-Agent with Qwen3.5-9B from 65.0 to 4.0, and Qwen-Agent with Qwen3.5-27B from 60.0 to 6.5). In the naturally occurring-conflict setting, some configurations have significantly lower accuracy on the conflict split than on the control split. However, on BrowseComp and part of GAIA, agents perform better on instances they hold false knowledge of. The authors note that the naturally occurring-conflict and control splits contain different questions and may differ in task difficulty.

• Higher accuracy is associated with lower Solve and Escalate rates. Across (agent, dataset) cells, the fitted curve declines from a Solve rate near 45% at low accuracy to under 10% at high accuracy, and from an Escalate rate near 38% to under 10% over the same range. Solve and Escalate are negatively associated with task accuracy across the plotted cells, while Identify shows no clear monotone relationship with accuracy.

• Some high-accuracy configurations do identify conflicts but do not communicate them. Claude Code reaches an Identify F1 of 90.0% on BrowseComp, yet its incorrect final answers often do not acknowledge unresolved uncertainty. OpenHands on GAIA reaches 51.0% accuracy on the conflict split but tends to ignore the conflict for instances it gets wrong. The authors suggest this may be because raising concern can be implicitly penalized: an uncertain answer could still be graded correct, while escalating will be graded incorrect.

• Conflicts are noticed early but rarely followed up. Conflict-relevant answer mentions peak in the first 10% of the trajectory and then fall sharply, following a long tail of additional steps. Both tracked modes (mentions of the ground-truth answer in trajectories that end incorrectly, and mentions of the agent's parametric answer in trajectories that end correctly) peak in that first 10%.

• A single system-prompt clause raises escalation but often lowers accuracy. On MoNaCo, Escalate rises from 1.6 to 60.7 for GPT-5, from 13.9 to 60.8 for Claude Code, and from 16.7 to 44.9 for Qwen-Agent (Qwen3.5-9B), while GPT-5 accuracy falls from 30.1 to 23.2. On BrowseComp, GPT-5 gains 28 points of Escalate but loses 6 points of accuracy. Most (agent, dataset) cells land in the "humble-but-inaccurate" region where prompted runs have higher Escalate and lower task accuracy than default runs. The reported significance test compares conflict with control, not prompted with default runs.

• Judge reliability and human agreement. The three judges agree with Gwet's AC1 of 0.93 to 0.97 on correctness and 0.82 to 0.92 on Identify and Escalate across every judge pair. Three human annotators, labeling 50 Identify turns and 50 Escalate final answers while blind to the judge label, agent identity, and split, agreed with each other on 76.0% of Identify turns and 85.3% of Escalate answers.

Methodology in Plain English

The authors start by asking each backbone model the same question zero-shot in two ways (free-form and multiple-choice), and keep only questions where the two modes agree and the answer is in the candidate set, so they know the model holds a confident parametric belief. They then build tasks where that belief is placed in tension with contextual evidence. For the controlled setting, each retained instance becomes a question plus two passages with mutually contradicting answers, and each is matched with a control version that has the same question and the same number of passages but consistent, ground-truth-aligned evidence. For the naturally occurring setting, they use a two-pass procedure (closed-book answer, then self-verification) to select tasks from GAIA, MoNaCo, and BrowseComp on which the backbone holds a confident but wrong parametric belief, and build a matched agentic control set from instances where the model is confidently correct or has no strong prior.

They then run each agent system on both conflict and control splits and record the full trajectory of reasoning traces, tool calls with arguments and results, and text turns shown to the user. All agents are capped at a maximum of 50 iterations. Final answers are graded both by rules (normalized exact match for single-reference instances, substring match for multi-reference instances, GAIA quasi-exact-match normalization, and a per-instance similarity for MoNaCo scalar answers) and by a GPT-5 judge that sees only the prediction and the reference, never the original question, so it cannot re-answer on its own.

Epistemic humility is scored from the trajectory rather than from one holistic judge call, so each facet can succeed or fail independently. A GPT-5 judge reads only the intermediate turns to produce the Identify label and the turn index where recognition first appears. Solve checks whether, after flagging a gap, the agent made at least one call after recognition with no exact duplicate and terminated cleanly without hitting max turns. Escalate checks whether the final assistant turn acknowledges residual uncertainty, computed over conflict instances and conditioned only on the gap not having been closed, not on Identify. The paper states that exact per-split instance counts are not reported in the provided content; the reported cost of the experiments was more than $3000 and more than 1000 GPU hours.

Why This Matters

For research, the paper argues that outcome-only scoring hides behaviors that are orthogonal to correctness: two agents with the same accuracy can differ substantially in whether they notice uncertainty, attempt to resolve it, and communicate it. It offers a concrete, released framework (per-turn ISE judgments plus full trajectories) for uncertainty-aware scoring and for supervising intermediate steps rather than only final answers.

Real-world applications:

• Coding assistants, where an agent may proceed confidently on a wrong assumption about a codebase despite evidence in retrieved files or tool output.
• Research agents, where retrieved sources may contradict each other or contradict the model's prior, and the user needs to know which claims are unsettled.
• General-purpose AI assistants that act over multiple steps with search and code execution, where a conflict raised mid-trajectory can be silently overwritten by the next tool call.
• Tasks where failing silently is costly, since the paper's framework motivates giving agents an explicit abstain-or-escalate action so an honest admission is not systematically worse-scored than a confident wrong answer.

Industry relevance: the paper reports that on the controlled-conflict benchmark, configurations lost more than 30 percentage points of accuracy relative to no-conflict controls, and that a one-line prompt change raised escalation while often lowering accuracy. The authors conclude that epistemic humility emerges from the interaction among the backbone model, the agent harness, and the evaluation environment, and that their experiments do not isolate the respective contributions of those three.

Future Directions

• Factorial backbone × harness comparisons. The authors ran five configurations with a fixed backbone per configuration and did not run a full backbone × harness combination due to budget constraints, so the causal contributions of model, harness, and benchmark remain untested.

• Harness-level and training-level interventions. Explicitly studying a first-class abstain-or-escalate action in the harness, and process-supervised fine-tuning for honest abstention, rather than only a single system-prompt clause.

• Trajectory-level analysis of where conflicts get dropped, and which harness affordances and scoring choices would preserve and reward uncertainty acknowledgment.

• Broadening beyond knowledge conflict. The ISE framework does not cover other facets of humility, such as resisting user pressure to agree, declining tasks for which the agent lacks the right tools, expressing calibrated confidence over continuous quantities, or handling ambiguous instructions and genuinely unknowable questions.

Target Audience

Agent researchers and evaluation designers who build or benchmark LLM agent systems; practitioners deploying multi-step agents with tool use who care about failure modes other than final-answer accuracy; and readers interested in measurement modeling of abstract constructs such as honesty, calibration, and humility in computational systems. The paper is most useful to those already familiar with agent harnesses and benchmark scoring, but its central argument is stated plainly enough for newcomers to agent evaluation.

Authors’ abstract

When retrieved evidence contradicts an agent's prior beliefs, does it revise its answer, acknowledge uncertainty, or persist with an incorrect conclusion? Existing evaluations of agentic systems focus primarily on task success, offering limited insight into how agents handle such conflicts. We propose to evaluate agents on epistemic humility (EH): the agent's willingness to recognize, act on, and communicate uncertainty during task execution. We operationalize EH through three trajectory-level behavioral dimensions: Identify, Solve, and Escalate (ISE). Through knowledge conflict, situations where the backbone language model's parametric knowledge contradicts the evidence it encounters, or where two contextual sources disagree, we evaluate two conflict settings: (1) controlled conflict and (2) naturally occurring conflict during multi-step agentic execution, each paired with matched no-conflict controls. Evaluating four agents, we find that higher task accuracy does not necessarily correspond to greater epistemic humility: some high-accuracy configurations recognize conflicts during execution but do not communicate unresolved uncertainty in their incorrect final answers. Trajectory-level analysis further reveals that agents frequently detect conflicts in early steps of execution but fail to maintain or resolve them in later steps. Finally, we show that model-level interventions can improve EH, but often at the cost of task accuracy, suggesting that epistemic humility emerges from the interaction among the backbone model, agent harness, and evaluation environment.