The Pulse
Apple Finds Tool Search Should Change With Task Complexity
Apple researchers compared two ways for AI agents to find APIs and found that the better method depended on whether a request needed one tool or several. Their study tested the approaches against nearly 47,000 API endpoints and reports a tr
AI.info Team · Written with AI ·
Apple researchers’ comparison of two ways for AI agents to find tools turns on a simple distinction: whether a task needs one API or several. In a paper submitted to arXiv on October 13, they report that searching directly from the user’s query performs better for single-tool requests, while asking a language model to sketch needed capabilities first helps with multi-tool requests. The paper, accepted to EMNLP 2026’s Industry Track, tests both approaches against nearly 47,000 API endpoints.
Two routes into a 46,984-API registry
The first method, Retrieve-then-Plan (RtP), embeds the user’s query and uses nearest-neighbor search to return matching API descriptions. The agent then plans with those results. The alternative, Plan-then-Retrieve (PtR), first asks a model to draft “Abstract Tool Stubs”: hypothetical function signatures that describe capabilities the task may need. The system searches the registry with those stubs, then gives the matching tools to the planner.
Apple’s team evaluated the methods on ToolBench, using 46,984 unique API endpoints across 50 categories. Its test set separated single-tool requests (G1), multi-tool requests within one category (G2), and multi-tool requests spanning categories (G3). The main comparison used a retrieval budget of five results; the researchers also varied that budget from one to 20 and tested registry sizes from 1,000 to 47,000 APIs.
Direct search wins on single-tool requests
On the 200 single-tool G1 queries, RtP recorded recall of 0.372, against 0.289 for PtR, and a higher pass rate: 0.884 versus 0.840. The authors say direct retrieval has an advantage when a query contains a recognizable API name or brand. In those cases, the planning step can lose a useful clue by replacing the specific name with a more general description.
The outcome reverses on the 200 G2 multi-tool queries. PtR reached recall of 0.188, compared with 0.143 for RtP, and a pass rate of 0.855 versus 0.747. Breaking a request into capability-level stubs can give retrieval separate searches for its component needs, where a single query embedding may not represent them all.
For the 100 cross-category G3 queries, PtR’s recall was higher—0.172 versus 0.160—but the difference was not statistically significant. Its pass rate was 0.820, against 0.732 for RtP. The paper cautions that G3’s smaller sample gives that recall comparison less statistical power.
Invented tool names can help—or mislead
PtR’s stubs are not necessarily real APIs. The model invents plausible tool names and descriptions, and the researchers identify this as a mixed effect: generic stubs can discard proper-noun clues that help RtP on simple tasks, but can also split complex requests into functional sub-needs. The paper says only 40.6% of G1 stubs and 24.8% of G3 stubs matched a relevant API at all.
“Together, these results confirm that the crossover reflects structural properties of query-registry coupling rather than search breadth alone.”
Mingfei Zhang and co-authors, the paper’s authors
The pattern held across retrieval budgets, registry sizes, and a replication using gpt-oss-120b alongside Gemini 2.5 Flash. That consistency supports the paper’s central finding, but the authors’ evaluation has limits: they use ToolBench’s reference API trajectories as a proxy for retrieval quality, rather than executing the APIs. The study does not assess parameter correctness or the quality of agents’ final answers.
An extra model call has a measurable price
PtR adds a discovery call before retrieval and planning. The paper’s analysis puts the extra token overhead at roughly 75% to 104%, and the sequential call also adds a round trip before the planner can start. That cost hurts on single-tool requests, the paper finds, but pays off on multi-tool requests.
The researchers propose routing requests between the methods: RtP for single-tool tasks and PtR for multi-tool ones. In their analysis, that combination cuts total inference tokens by about 16%—roughly 36% of PtR’s discovery-call overhead—without reducing retrieval quality relative to always choosing the better method for each subset. One possible routing signal is RtP’s own top-result similarity score, which is already computed; the paper leaves empirical validation of a routing classifier for future work.