Tool Contract Testing and Tool-Use Evaluation

Passing the schema test proves nothing about selection, arguments, or interaction. That is where tools usually fail. The last stage in the row — the state test — is the one an API suite structurally cannot reach. It asks about the world after the call, not about the call.

A published benchmark exists because of exactly that gap. Apple built ToolSandbox in 2024, against the previous generation of tool-use evaluations. Its abstract names what that generation missed: “While previous works focused on either evaluating over stateless web services (RESTful API), based on a single turn user prompt, or an off-policy dialog trajectory, ToolSandbox includes stateful tool execution, implicit state dependencies between tools, a built-in user simulator supporting on-policy conversational evaluation and a dynamic evaluation strategy for intermediate and final milestones over an arbitrary trajectory.”

Three categories come back as hard even for the most capable models: State Dependency, Canonicalization and Insufficient Information. One tool's output silently conditioning another. The same entity written two ways. A request that does not contain enough to act on. None of the three can even be expressed in a stateless single-call test. The interaction test and the state test need owners of their own. Nothing upstream of them is looking.

A good tool has to behave correctly as a service and be usable by an agent. A complete evaluation looks at each stage separately: how the agent discovers the tool, selects it, builds its arguments, executes it, interprets the result, recovers from an error, and verifies the final state.

Testing only happy-path calls misses ambiguity, policy conflicts, asynchronous completion, lost responses, stale state, and adversarial tool outputs.

The last item on that list has a case number and a score. Connecting to an untrusted MCP server could run arbitrary OS commands on the client machine. The affected npm package was mcp-remote, from version 0.0.5 up to but excluding 0.1.16. That is CVE-2025-6514, published 9 July 2025, with a CVSS 3.1 base score of 9.6, Critical. The advisory states the mechanism in one line: “mcp-remote is exposed to OS command injection when connecting to untrusted MCP servers due to crafted input from the authorization_endpoint response URL”.

Read what is hostile there. Not the user's request. Not the arguments the model built. The tool server's own response, arriving in a field the client was written to trust. Version 0.1.16 fixed it. A suite that replays only well-formed responses from a server you control cannot produce that finding. The input that breaks the client never appears in it.

Where an API unit test stops, an agent tool eval begins. Only an end-to-end task eval sees the whole run, and the natural-language interface is visible to just one of the three. What you want to see is the agent still picking the right tool when the catalog is as crowded as the real one. What you are guarding against is the call that returns success and is still the wrong thing to have done.

The drop between the second column and the third has been measured on real software. WebArena runs agents against fully functional websites in four domains and grades the functional correctness of what they complete — what the environment looks like at the end, not how the answer reads. It was published in 2023. Under that rule: “The results demonstrate that solving complex tasks is challenging: our best GPT-4-based agent only achieves an end-to-end task success rate of 14.41%, significantly lower than the human performance of 78.24%.”

14.41 percent against a human 78.24 percent is the size of the gap that component-level competence can hide. Every one of those runs was made of individual actions, most of them well formed. The column that grades the finished state is the only one that saw the difference.

That headline is not a hypothetical, and it does not need an anonymous vendor to stand behind it. τ-bench grades an agent by comparing the database state at the end of a conversation with an annotated goal state. It was published in 2024. That rule is indifferent to how confident the transcript sounds, and to how many calls came back without an error. Under it, state-of-the-art function-calling agents such as gpt-4o solved under 50 percent of tasks on a single attempt.

The backends were not broken. What broke was everything the backend test never looked at: which tool was chosen, on which record, at which turn, under which policy, and whether the environment was left in the condition the task actually described. τ-bench also introduced the pass^k metric, which asks the same task k times rather than once. It measures consistency instead of the existence of one good trajectory. That is the property a demo can never show you, and a rerun always can.

Realistic catalogs are where selection breaks, and the breakage has been measured with named models and named numbers. Salesforce AI Research published MCP-Universe in 2025: agents put against 11 real-world MCP servers across 6 domains, scored by execution-based evaluators. “Through extensive evaluation of leading LLMs, we find that even SOTA models such as GPT-5 (43.72%), Grok-4 (33.33%) and Claude-4.0-Sonnet (29.44%) exhibit significant performance limitations.”

43.72, 33.33 and 29.44 percent are what selection accuracy looks like when the tool space is large and unfamiliar, which the paper names explicitly as part of the difficulty. That is the measurement Signal 1 needs. A curated catalog of five well-named tools will not produce it. Whether argument errors are detected before side effects is a separate check, and it has to pass on its own. Two more signals need repeated or adversarial runs before they carry weight: that recovery behavior matches the declared error semantics, and that end-to-end success can be traced to specific interface strengths or failures.

Lay out an evaluation matrix for one tool, with the stages of use down one side and the failure classes across the top. The matrix is finished when it has a cell for the successful call that was still the wrong action, and when the claim about selection accuracy in a crowded catalog has a test behind it rather than an assertion.

Two of the steps are where teams under-build. Both have a published ratio to aim at.

Start with varying instructions. AgentDojo is an evaluation environment for agents that execute tools over untrusted data, published in 2024, and it reports its own contents precisely: “We populate the environment with 97 realistic tasks (e.g., managing an email client, navigating an e-banking website, or making travel bookings), 629 security test cases, and various attack and defense paradigms from the literature.” That is roughly six adversarial cases for every ordinary one. Most internal suites carry the opposite ratio. Its result is two-sided and worth holding together: existing prompt injection attacks break some security properties but not all, while state-of-the-art LLMs already fail many tasks with no attack present.

Then repeating runs. The eighth run is not bookkeeping. It is the difference between under 50 percent and under 25 percent in τ-bench's retail domain, on the same tasks, with nothing else varied.

Tool tests become regression assets. Re-run them when descriptions, schemas, models, policies, or surrounding catalogs change. Note how fast that clock runs. The Berkeley Function Calling Leaderboard passed through AST scoring, contributed enterprise functions, multi-turn interaction and agentic evaluation across four versions, and its public leaderboard was last updated 2026-04-12. A suite written against the first of those versions is not measuring what the fourth measures.

Carry the same two points into every review. A call that succeeded is not automatically a call that was right. And the claim that the agent picks well among many tools has to be backed by measurements from a catalog the size of the real one: 11 real-world MCP servers across 6 domains produced 43.72, 33.33 and 29.44 percent for GPT-5, Grok-4 and Claude-4.0-Sonnet.