When Users Are Happy but Agents Are Wrong: Multi-Dimensional Evaluation of Tool-Augmented Dialogue

AI summary

Overview

Research area: Natural Language Processing — evaluation of tool-augmented conversational AI (LLM agents that call external tools/APIs).

Technical level: Intermediate. Readers should be comfortable with LLM benchmarking, rubric-based LLM-as-judge evaluation, and reward models, but the paper's framing is accessible.

Scope: The paper introduces TRACE, a synthetic benchmark of 566 tool-augmented conversations spanning 26 error situations, and SCOPE, an evaluation framework that scores conversations on more than user satisfaction alone.

What This Paper Is About

Conversational AI agents that call external tools can fail in ways users never notice — for example, the agent reports "Email successfully sent" while the email never went out, and the user walks away happy. Existing evaluation methods measure either how well an agent calls tools or how satisfied the user appears, but not both together, so they miss these silent failures. The paper builds a benchmark of deliberately synthesized tool-use conversations containing such errors and a multi-dimensional evaluation framework designed to catch them.

Key Contributions

1. TRACE benchmark ("Tool-Response Assessment of Challenging Errors"): 566 high-quality conversations comprising 2181 agent-user turns, covering correct executions plus 26 distinct error cases. It is positioned as the first benchmark combining tool-use, user satisfaction, real-world API coverage, negative samples, and conversational (multi-turn) structure.
2. A four-dimension characterization scheme for tool-use dialogues: tool execution correctness (correct / incorrect due to agent / incorrect due to tool error), agent performance (appropriate / not appropriate), user satisfaction (satisfied / dissatisfied), and overall conversation success (POS / NEG). The authors derive eight plausible attribute combinations from these dimensions and manually expand them into 26 distinct situations.
3. SCOPE ("Structured Conversation Observation for Performance Evaluation"), an extension of SPUR that adds area discovery, area-specific rubric extraction, severity-based rubric weighting, and make-or-break criteria for critical errors.
4. Systematic evaluation of SPUR, G-Eval, and reward modeling against SCOPE on TRACE, including ablations isolating each SCOPE component and a gold-versus-silver subset analysis.

Main Findings

• All frameworks fall well short of ideal. Across every method tested, performance on TRACE remains far from a solved problem, which the authors present as evidence of the task's fundamental difficulty.
• SCOPE achieves the best overall F1: 76% with both Sonnet-3.5 and GPT-4.1. With Sonnet-3.5, SCOPE reaches 0.79 overall accuracy, a 17.4% relative improvement over SPUR, and 0.76 F1, an 11.8% improvement.
• Reward modeling wins on the hardest subset but loses overall. The Skywork reward model reaches 0.70 accuracy on hard negatives (cases where the user is satisfied but the conversation is NEG) versus SCOPE's 0.32 with Sonnet-3.5 and 0.42 with GPT-4.1. However, the reward model's restrictive behavior hurts it on easy cases (0.86 accuracy versus SCOPE's 0.93) and yields a lower overall F1 of 0.75.
• SPUR nearly fails on hard negatives. SPUR records 0.02 accuracy with Sonnet-3.5 and 0 (0.1 SD) with GPT-4.1 on hard negatives, despite competitive easy-case scores (0.90 and 0.87).
• G-Eval sits in the middle. G-Eval scores 0.37 (Sonnet-3.5) and 0.42 (GPT-4.1) on hard negatives, with overall accuracy of 0.77 and 0.76 respectively.
• SCOPE's rubrics generalize beyond user satisfaction. SCOPE autonomously extracts rubrics covering tool-use correctness and agent–tool interaction that align with error categories defined in the benchmark, even though SCOPE has no access to the synthesis process.
• Rubric weighting and make-or-break criteria drive most of the gains. Removing rubric weight estimation drops overall accuracy by 13.9% with Sonnet-3.5 and 37.97% with GPT-4.1; removing make-or-break weights drops it by 19% and 35.44% respectively. Removing area discovery changes accuracy by 0.01% (Sonnet-3.5, improving) and 0.05% (GPT-4.1, decreasing).
• Models perform slightly better on the silver subset than gold. SCOPE scores 0.80 accuracy on silver versus 0.76 on gold, a pattern seen across all models, which the authors attribute to the LLM judge's 93.13% precision filtering possibly missing nuanced cases that human annotators caught.
• GPT-4.1 generates more critical rubrics than Sonnet-3.5, improving hard-negative detection (0.42 vs. 0.32) while keeping easy-case performance comparable (0.90 vs. 0.93).

Methodology in Plain English

The work has three stages.

Defining what "wrong" means. The researchers first describe four attributes of a tool-use conversation: whether the tool executed correctly and whose fault it was if not, whether the agent behaved appropriately, whether the user was satisfied, and whether the whole conversation should be labeled POS or NEG. Critically, overall success is not the same as user satisfaction — an agent that falsely claims a tool succeeded gets a NEG label even if the user is pleased. From the plausible combinations of these attributes they manually write 26 concrete situations, such as "user unsatisfied + tool error + appropriate agent behavior," which covers the agent suggesting alternatives for an inaccessible tool.

Generating conversations. For each of the 26 situations, an LLM writes a multi-turn user–agent conversation in which the tool calls are also simulated by the LLM. Tools were drawn from existing datasets: 28 from ToolTalk, 1 from MINT, and 1 from API-Bank, organized into nine groups (account, alarm, calendar, email, message, reminder, weather, reasoning, api_bank). NEG conversations were generated with one-shot prompting (with manually curated examples) because they have stricter requirements, while POS conversations used zero-shot generation for diversity. Claude Sonnet-4 and DeepSeek-R1 were the generators.

Filtering for quality. Two rounds of validation followed. First, human annotators — a crowd-sourced group and three domain experts — judged whether each conversation exactly matched its case description, achieving 80.7% agreement (Cohen's Kappa 0.385). This produced the gold set. Second, a Sonnet-4 LLM judge was calibrated on the gold set and reached 93.13% precision; conversations passing it formed the silver set. Together these yield 566 conversations (182 POS, 384 NEG).

Evaluating. SCOPE extends SPUR's pipeline by first discovering evaluation areas from training conversations, then extracting reasons and rubrics per area, assigning each rubric a weight from 1 to 10 (with a special weight of 100 for make-or-break negative criteria), and finally labeling a conversation by comparing average positive and negative scores against a threshold of 0. All frameworks were tested with 5-fold cross-validation over a 40/60 train/test split (226 training samples, 340 test samples), using Claude Sonnet-3.5 and GPT-4.1 as backbones, at a total experiment cost of roughly 200 USD.

Why This Matters

Impact on research. The paper argues that user satisfaction signals — the dominant proxy in dialogue evaluation — are actively misleading for tool-augmented systems. It provides both a labeled resource with controlled error types and evidence that severity weighting (rather than treating all criteria equally) is the mechanism that enables detection of silent failures. It also establishes a benchmark on which current state-of-the-art methods demonstrably fail, giving the field a measurable target.

Real-world applications:

• Customer-support agents that report ticket resolution while the underlying system action silently failed.
• Scheduling and productivity assistants that confirm a calendar event, reminder, or alarm that was never actually created.
• Email and messaging agents that claim a message was sent when it was not.
• Any tool-integrated assistant where an undetected wrong parameter or unhandled API error produces a confident but false confirmation to the user.

Industry relevance. Organizations deploying tool-calling agents need automated quality monitors that do not rely on whether users complained. Because dissatisfied users only surface a fraction of failures, satisfaction-based monitoring undercounts errors — a direct operational risk for teams using conversation analytics to gate model releases.

Future Directions

• Move beyond synthetic data. The authors note the benchmark is synthetic and propose extending TRACE with naturally occurring conversations or hybrid synthetic–real corpora for greater ecological validity.
• Use real tool execution. Tool execution is currently simulated inside the generated conversation; the authors suggest either issuing actual API calls or building a dedicated tool execution simulator capable of tool-specific errors.
• Scale beyond 30 tools and beyond English. The dataset is limited to the 30 tools drawn from existing datasets and to English, leaving cultural and language-specific evaluation issues unaddressed.
• Improve hard-negative detection. SCOPE's hard-negative performance (0.32 with Sonnet-3.5, 0.42 with GPT-4.1) remains far from perfect, and the authors explicitly hope follow-up work targets exactly these cases.

Target Audience

Researchers and practitioners working on LLM agent evaluation, tool-use benchmarks, and LLM-as-judge or reward-model-based conversation scoring. It is also relevant to product and quality teams at organizations shipping tool-augmented assistants, who need evaluation signals that go beyond user satisfaction, and to dialogue-systems researchers interested in multi-dimensional annotation schemes and severity-weighted rubrics.

Authors’ abstract

Evaluating conversational AI systems that use external tools is challenging, as errors can arise from complex interactions among user, agent, and tools. While existing evaluation methods assess either user satisfaction or agents' tool-calling capabilities, they fail to capture critical errors in multi-turn tool-augmented dialogues-such as when agents misinterpret tool results yet appear satisfactory to users. We introduce TRACE, a benchmark of systematically synthesized tool-augmented conversations covering diverse error cases. Evaluation with state-of-the-art conversation evaluation frameworks reveals that all approaches remain far from ideal performance, demonstrating the fundamental difficulty of this benchmark.