CRM-Agent-Bench: Benchmarking Tool-Using Agents in Stateful Enterprise Environments

AI summary

Overview

Research area: Human-Computer Interaction / evaluation of tool-using language-model agents in enterprise software (customer relationship management).

Technical level: Intermediate. The framing is conceptual and readable, but the paper uses formal notation (partially observable Markov decision processes, conjunctive contracts) and assumes familiarity with agent benchmarks such as τ-bench and ToolSandbox.

Scope (one sentence): The paper introduces CRM-Agent-Bench, a 555-task, nine-domain benchmark that grades complete agent systems on whether they actually execute multi-step CRM business transactions correctly, within declared policy limits, and at what latency and cost.

What This Paper Is About

When a language agent is given authority to change records in a business system, an incorrect answer is no longer the worst failure: the agent can complete the wrong transaction, keep going after a backend rejection, or claim an operation succeeded that never ran. Existing CRM benchmarks largely measure retrieval and answer correctness rather than whether the underlying state change happened correctly and within policy. The authors build a benchmark where every task carries its own executable "contract" of required actions, arguments, ordering, responses, target state, and protected state, and an agent passes a task only if every declared check passes at once.

Key Contributions

1. An execution-grounded benchmark of 555 stateful, multi-turn tasks spanning nine business domains and 36 capability tags, drawn from five CRM applications developed with practitioners and from 22 tool-using workflow families adapted from a public, versioned collection of agent recipes. The benchmark runs without a live CRM or vendor backend because all records and transactions are synthetic.

2. Task-specific conjunctive contracts. Each task declares only the checks relevant to it, drawn from six families (expected actions, argument constraints, ordering, response content, target state, and forbidden actions/arguments/state). Success is the strict product of those checks, so a persuasive reply cannot offset a missing transaction and a correct endpoint cannot erase a forbidden intermediate action. For the response-content family only, a fixed LLM judge sees just the user instruction and visible replies and cannot override the programmatic checks.

3. An agent-agnostic evaluation protocol that reports four facets separately rather than collapsing them into one score: outcome, trajectory, operational viability (latency and cost), and policy compliance. It also includes a static capability-exposure audit that checks whether each forbidden tool is even reachable in its domain graph — important because a forbidden action may be absent because the model refused, because it failed to act, or because routing never exposed the tool.

4. An empirical comparison of sixteen systems with the full leaderboard, per-domain breakdowns, efficiency trade-offs, and a three-round reliability analysis, showing that systems which look nearly identical on a single headline accuracy number diverge sharply on cost, per-domain reliability, and difficulty scaling.

Main Findings

• Strict success spans a 19.28-point range. Across the sixteen full-suite runs on the audited 555-task set, strict accuracy runs from 83.60% (Gemini 3.8 Flash) down to 64.32% (Nemotron 3.5 Lightning 30B and GPT-4.1 Nano, which tie). All runs produced valid grades for all 555 tasks with zero infrastructure exclusions.

• The leaderboard is tightly compressed at the top. Eight models (ranks 1–8) fall within a 2.88-point band, from 80.72% to 83.60%, and three tie exactly at 81.26% (451/555): Claude Sonnet 5, GPT-5.5, and GPT-5.6 Sol. A single headline number therefore leaves the leading systems nearly indistinguishable.

• Response-only grading inflates success by 10.5 to 11.0 percentage points. For the four systems analyzed, adding trajectory and state checks to a response-only grade removes a large, consistent slice of apparently successful tasks; the action check alone removes 45 to 50 of the 555 tasks. Semantic-response pass rates stay high — 92.3% for the strongest system and above 77% for every run — while the action check passes only 70.3% to 89.8% of applicable tasks and the argument check 70.2% to 90.7%.

• Action mismatch is the biggest failure category. Under the paper's primary-label precedence, action mismatch is the single largest non-success category for every one of the sixteen runs, from 54 tasks for the strongest system to a maximum of 157 for Nemotron Super 3 120B; semantic-response, argument, and negative-check failures are individually much smaller.

• No model dominates every domain. Customer support is by far the hardest domain: its across-model mean strict success is 56.35%, 20.80 points below the next-hardest domain (travel, 77.15%), and it is the largest domain at 125/555 tasks. Onboarding has the highest across-model mean (90.24%). Gemini 3.8 Flash, the overall leader, is strongest only on retail (87/95); travel goes to Inkling Small NVFP4 (78/93) and ITSM to GPT-5.6 Luna (63/66). Muse Glimmer 30B is strongest on customer support (85/125, 68.00%) despite ranking 12th overall at 78.74%. GPT-5.6 Sol is the only model to sweep sales (18/18) and ties for strongest on field service (23/23, with Claude Sonnet 4.6).

• Equal headline scores can hide different failure profiles. The two lowest-ranked models tie at 64.32% but get there differently: GPT-4.1 Nano is comparatively strong on retail (77/95, 81.05%) and has the single lowest domain score in the heatmap on customer support (40/125, 32.00%), while Nemotron 3.5 Lightning 30B is more evenly weak.

• Difficulty scaling is the clearest headroom signal. Strict success declines monotonically from easy to hard tasks for every system, with a within-model easy-to-hard drop of 5.37 to 27.39 points.

• Cost varies far more than accuracy and does not track rank. Across the thirteen priced runs, estimated tested-agent cost spans 89.3×, from GPT-4.1 Nano at $2.14 to Claude Opus 5 at $191.00 — the latter only 2nd on accuracy. Gemini 3.8 Flash is the informative reference: it leads on accuracy at $16.12, the fifth-cheapest priced run, and no priced model costing as much or more scores higher. Claude Haiku 4.5 costs $24.50 more than Gemini 3.8 Flash while scoring 7.56 points lower. Gemini 3.8 Flash also has the lowest mean tested-agent token count (24.0k per scored task), making it the sole Pareto-optimal point in the token panel. Three runs (Inkling Small NVFP4, Muse Glimmer 30B, Nemotron 3.5 Lightning 30B) are self-hosted open-weight checkpoints without metered per-token prices, reported as "n/a."

• Latency and tool budgets diverge widely. Nemotron 3.5 Lightning 30B has the lowest mean LLM latency (5.86 s) and Inkling Small NVFP4 the second-lowest (6.66 s), but they score 64.32% and 79.64%. Gemini 3.1 Pro Preview and Muse Glimmer 30B sit at the high-latency extreme (69.08 s and 84.96 s). Tool calls per task differ nearly four-fold: Nemotron Super 3 120B issues the fewest (2.09) and Nemotron 3.5 Lightning the most (7.68).

• Avoiding prohibited behavior and completing the permitted workflow are distinct dimensions. Of 555 tasks, 390 have no negative assertion and 165 are policy-constrained. Across 2,640 policy-constrained model–task trials, 2,408 (91.2%) pass all applicable negative checks but only 1,764 (66.8%) pass the full contract — so 644 trials (24.4% of the subset, 73.5% of its 876 strict failures) avoid violations yet fail another required component. Strict success on tasks without negative assertions is 82.1%.

• Results are mostly stable but not identical across repeats. In three rounds, Gemini 3.8 Flash ranks first every round with mean strict success of 85.53% (sample SD 1.66 points); Claude Opus 5 averages 83.29% (SD 0.62) and stays second. Pairwise Spearman rank correlations range from 0.869 to 0.934. Yet task-level outcomes switch: Gemini passes 480 tasks in both rounds 2 and 3 but 30 task outcomes flip between those executions, and only 434/555 tasks succeed in all three rounds (pass³ = 78.20%) versus the 85.53% mean success rate.

Methodology in Plain English

The authors treat each task as a small controlled world. A task specification pairs a private user goal, an optional initial-state overlay, a domain with typed tools, and a contract. Each trial starts from a freshly reset synthetic CRM environment.

Instead of replaying a fixed transcript, a simulated user reads only the agent's user-visible replies and generates the next turn from the task goal and conversation so far. The simulator never sees the agent's private reasoning, tool calls, tool results, routes, or database state, and it emits a stop marker when it judges the interaction complete. Otherwise the trial runs to a 30-turn cap or a 1,200-second timeout. The reference protocol uses gpt-5-mini as the user simulator.

The system under test is a routed agent: on each user turn a graph-structured reference scaffold selects one capability node, which acts through tools scoped to that node. Changing the router or the exposed tool catalog changes the access policy but not the task or its contract. When the interaction stops, the harness freezes four evidence channels — visible dialogue, ordered route log, complete tool trace, and terminal state — and applies only the checks that task declared.

Grading is conjunctive: success is the product of the declared binary checks. Expected-action checks match successfully executed calls, and forbidden-action checks reject prohibited attempts including calls the tool itself rejected. Ordering uses subsequence or exact filtered-sequence matching. State checks address sparse paths with exact values, non-null requirements, and minimum collection lengths. Forbidden-argument checks block calls carrying another party's identifiers.

Failures are separated by attribution: model-attributable failures (hallucinated tools, verified non-convergence at timeout) count as zero, while infrastructure failures are excluded from the scored denominator. The paper reports scored coverage — the fraction of scheduled tasks yielding a valid grade — alongside conditional success, plus latency and cost for the tested agent model, rather than a single composite index. All task identifiers and configuration were frozen before model comparison, and every task passed schema, tool-reference, and fresh-environment gold-path validation.

Why This Matters

Impact on research. The paper argues that a deployable-agent benchmark must jointly assess outcome, trajectory, operational viability, and policy compliance, and shows empirically what goes wrong when it does not: response-only grading overstates success by roughly 11 points, and negative-check compliance (91.2%) decouples from full-contract completion (66.8%). It supplies an auditable, self-contained artifact with per-check component evidence, so failures can be localized rather than summarized away, and it explicitly warns that measured safety can be confounded with general task capability.

Real-world applications:

• Selecting an agent for a CRM deployment, where accuracy at $16.12 and accuracy at $191.00 are not the same purchase decision.
• Pre-deployment regression testing for orchestration scaffolding, tool inventories, and routing policies, since the same base model behaves differently under a different tool inventory.
• Designing guardrails and capability exposure, because a forbidden action may be absent only because routing never exposed the tool.
• Auditing policy-constrained workflows such as refunds, transfers, and coupon application, where the gap between saying and doing is a direct financial loss.

Industry relevance. The benchmark is built from real CRM workflow structure and needs no live vendor backend, which makes it usable as an internal gate. The results argue for multi-axis agent selection in enterprise SaaS — weighing execution cost, per-domain reliability, and policy-constrained completion alongside strict accuracy — rather than ranking on one number.

Future Directions

• More controlled repetitions and human/domain-expert validation. The authors note only one execution per task on the main leaderboard and report descriptive comparisons rather than significance claims; three rounds are reported in the appendix, and they state they are actively extending the setting.
• Broader modalities and settings. Document-rich, visual, voice, and live-service workflows are named as planned extensions, which would test robustness and external validity while keeping executable, auditable grading.
• Treating tool exposure and routing as first-class design variables. Because turn-level exposure depends on routing, separating "the model refused" from "the model failed to act" from "the tool was never exposed" remains an open measurement problem the paper flags rather than solves.
• Simulator sensitivity. The reference protocol fixes gpt-5-mini as both user simulator and semantic judge, and sensitivity to simulator choice is deferred to an appendix; how much measured success moves with a different simulator is a natural follow-up.

Target Audience

Agent researchers and benchmark designers who need to measure executed behavior rather than answer quality; ML and platform engineers building tool-using assistants over mutable systems of record; enterprise product and reliability teams choosing between agent systems on cost, per-domain reliability, and policy compliance; and safety researchers interested in why the absence of a forbidden action is not by itself evidence of sound behavior.

Authors’ abstract

Deploying language agents in real applications demands more than the right answer: an agent must execute a multi-step business process correctly, follow domain-specific rules, and interact with a user across a dialogue, where the gap between saying and doing is the gap between a satisfied customer and a real loss. We argue that a deployable-agent benchmark should jointly assess four facets: the outcome an agent reaches, the trajectory it takes, its operational viability, and its policy compliance. Customer relationship management (CRM), where mutable records meet operational rules and consequential effects, exercises all four. We introduce CRM-Agent-Bench, an execution-grounded benchmark of 555 stateful, multi-turn tasks across nine business domains and 36 capability tags. Each task supplies a task-specific executable contract drawn from checks over actions, arguments, order, response, target state, and protected state; success is the strict conjunction of the checks declared for that task. We evaluate the complete agent system through these contracts and report scored coverage, latency, and cost. Across sixteen systems evaluated on the audited final release, strict success spans 64.32% to 83.60%; the closely grouped leading systems nonetheless diverge sharply in cost and per-domain reliability. CRM-Agent-Bench offers an auditable framework for measuring contract-grounded business-transaction integrity and adherence to task-declared benchmark policies, with a portable structure designed for extension as models improve. Github repository: https://github.com/salesforce-misc/crm-agent-bench