Tool Schema Design and Agent Affordances

A tool schema is part of the agent's action space and documentation. Names, descriptions, field constraints, defaults, and result envelopes shape which actions the model considers and how it interprets outcomes.

Good affordances reduce ambiguity rather than compensating for it with longer prompts. Separate actions that differ in consequence, authority, latency, or reversibility.

The description is not documentation. It is what the model reads at the moment it decides. Anthropic's tool-use documentation says so: “Claude determines when to call a tool based on the user's request and the tool's description”. Leave a required parameter out and nothing stops. Claude “might also infer a reasonable value”.

Schema design has hard edges too. Microsoft's guidance on structured outputs is blunt about them. “All fields must be required”. “Always set additionalProperties: false in objects”. And a ceiling on the whole thing: “A schema can have up to 100 object properties total, with up to five levels of nesting.”

How much weight that leaves on the description has been measured. MetaTool collected names and descriptions for 390 tools and built ToolE, a dataset of 21,127 user queries. That was in 2023. It split tool selection into four subtasks: similar choices, specific scenarios, possible reliability issues, and multi-tool. Across eight evaluated LLMs, selection stayed unreliable. The recommendation the paper closes on is not aimed at the model: “we strongly recommend that tool developers choose an appropriate rewrite model for generating new descriptions based on the downstream LLM the tool will apply to.” Twenty-one thousand queries later, the artifact to fix is the sentence a developer wrote about the tool.

Purpose, arguments, consequences, results — a tool schema reads in that order. So do its defects, and the order is not cosmetic.

That order has a price you can look up. A 2024 study measured it directly. For gpt-4o-mini-2024-07-18 it reports exact-match accuracy on GSM8K of 94.57 in natural language against 86.95 in JSON mode. On Last Letter Concatenation the same comparison is 83.11 against 76.00. The cause of the Last Letter collapse was which key the schema invited the model to emit first: “Upon inspection, we found that 100% of GPT 3.5 Turbo JSON-mode responses placed the “answer” key before the “reason” key, resulting in zero-shot direct answering instead of zero-shot chain-of-thought reasoning.” Not most responses. All of them.

A field order is therefore a behavior, not a formatting preference. Read the schema in its own order, and give results and errors different owners and different tests.

A tool, in the Model Context Protocol, is a name, an optional title, a description, an inputSchema, an optional outputSchema and optional annotations. Tools there are model-controlled: the model discovers and invokes them from its own reading of the context. Consequence is declared in the annotations. Revision 2025-06-18 of the specification is explicit about how far that declaration goes: “For trust & safety and security, clients MUST consider tool annotations to be untrusted unless they come from trusted servers.”

A shipped client acts on the hint anyway. Microsoft 365 Copilot drops the per-use confirmation dialog for any tool whose tools/list response sets readOnlyHint to true. The initial first-time prompt still appears. The dialog that would otherwise follow it does not. Microsoft Learn puts it plainly: “Developers can control whether Microsoft 365 Copilot asks the user for confirmation … by setting the readOnlyHint property to true”. It adds that “any action with side effects in the external system should be marked with true to ensure the user is in control”.

So the schema does not merely describe the action. One field, filled in by the tool's author, decides whether a person sees the action before it runs. The protocol tells clients not to trust that field. The model reaching for the tool reads the same metadata as its instructions.

Measure “Agents select the intended tool under paraphrased instructions” first, because paraphrase is what real traffic looks like. “Invalid parameter combinations cannot pass schema validation” is checked on its own. Two further claims hold up only under repetition and attack: “Results contain the evidence needed for downstream decisions” and “Tool descriptions remain accurate when backend behavior changes”.

Repetition has a published number. τ-bench introduced pass^k in 2024. The metric requires the same task to succeed on all k independent trials: “Our experiments show that even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail).” Under 50% once. Below 25% eight times in a row. That spread is the distance between a demo trace and a tool you can route traffic through.

For selection and validity there is a public leaderboard to measure against. The Berkeley Function Calling Leaderboard, published at ICML 2025, evaluates function calls with an Abstract Syntax Tree method that scales to thousands of functions. It also scores not calling as an action: “Finally, BFCL benchmark evaluates the ability of models to abstain and reason in stateful multi-step agentic setting.” Restraint stops being an unmeasured virtue. Its finding maps the work that is left. State-of-the-art LLMs handle single-turn calls well. Multi-step decision-making stays open.

Review one tool the way a product team reviews a screen. Read its name and its fields as the only instructions anyone will get. For a model-controlled tool that is literally true. The review should surface the places where poor affordances make invalid behavior look reasonable: a field whose name invites the wrong value, an option that reads as harmless and is not, a key whose position changes what the model does before it answers.

Two of the steps below have already been done once under real consequences. After 28 February 2017 AWS constrained the capacity-removal command so it removes capacity more slowly. It also removed the hidden mode in which a subsystem could be taken below its minimum required capacity level. Neither fix was a better instruction to the operator.

Then reword a handful of requests and check that the agent still reaches for this tool rather than a neighbouring one. Check it the way τ-bench does: on all k trials, not on the one that went well.

A high-quality tool can improve reliability more than another reasoning prompt. It narrows the action space and makes failures legible to both model and operator. MetaTool reached that conclusion after 21,127 queries over descriptions collected for 390 tools. Its recommendation went to the people who write the descriptions, not to the model reading them.

So design the tool against the failure where bad behavior looks reasonable to the model. Then settle the question with paraphrase. Ask for the same thing in different words, repeatedly. Once is not a result: gpt-4o solved under 50% of τ-bench's tasks outright and fell below 25% at pass^8 in the retail domain. If the agent still picks this tool, the design held. If it does not, the schema is what needs changing, not the prompt.