The Pulse
Salesforce Koa beats GPT-4.1 on tool-use benchmarks
Salesforce’s Koa model outperforms GPT-4.1 on two tool-use benchmarks, according to the company’s technical report, while trailing stronger frontier models in broader comparisons.
AI.info Team ·
Salesforce’s new Koa model beats OpenAI’s GPT-4.1 on several tool-use evaluations, but the company’s own paper places it below the strongest frontier systems. The results frame Koa less as a general-purpose model replacement than as an attempt to specialize an open-weight foundation model for enterprise workflows.
Salesforce describes Koa in a technical report posted to arXiv on September 14, 2026. The model starts with Nemotron-3-Super-120B, an open-weight foundation model, and receives additional training through Group Relative Policy Optimization, a reinforcement-learning method known as GRPO.
“On Tau2Bench it reaches a task-weighted average of 69.41, edging its Nemotron base (68.64) and outperforming GPT-4.1 by 14.9 points.”
Zixiang Chen, a co-first author of the report at Salesforce Agentforce & AI Research, and the paper’s co-authors wrote.
The authors say they use public and synthetically generated interactions rather than customer data. Salesforce developed the system with NVIDIA, according to the report.
Koa’s gains come from workflow specifications
Koa’s main distinction is the way Salesforce constructs training tasks. For enterprise scenarios, the company uses Agent Script, its declarative language for building Agentforce agents. Agent Script specifications describe routing, specialized subagents, typed actions, available tools and workflow instructions.
A simulation pipeline converts those specifications into multi-turn sessions shaped by different scenarios and user personas. The resulting environments expose the model to tool calls, user responses and workflow state. A separate helper model can act as the simulated customer, produce tool outputs where no production backend exists and judge whether the task was resolved.
Rewards depend on successful completion rather than only on matching a reference answer. Salesforce says the judge checks whether each sub-question has been resolved and requires customer- or system-specific answers to be grounded in a successful relevant tool call. In a separate deterministic sandbox, the reward is one only when the model’s actions reproduce the target end state.
GPT-4.1 falls behind on two public tests
The report evaluates Koa on Tau2Bench, BFCL and CRM Bench. Tau2Bench measures multi-turn customer-service interactions across airline, retail and telecom tasks; BFCL evaluates tool calling across multi-step, web-search, memory and stateful-tool settings; CRM Bench covers Salesforce and Agentforce workflows.
Koa records a 69.41 task-weighted average on Tau2Bench, compared with 68.64 for its Nemotron base and 54.48 for GPT-4.1. On BFCL, Koa scores 66.63%, ahead of the base model’s 64.73% and GPT-4.1’s 53.96%.
The CRM results are closer. Koa reaches an overall score of 0.86, compared with 0.84 for Nemotron-3-Super-120B, 0.81 for GPT-4.1 and 0.87 for Claude Opus 4.8. Koa’s function-call accuracy is 0.77, up from 0.71 for its base model, while its overall CRM result remains just below Opus 4.8.
Open weights bring control, with a performance ceiling
Salesforce presents the results as evidence that specification-driven reinforcement learning can adapt an open-weight model to enterprise agent work. The approach also gives the company a way to tie agent authoring and model training together: the same Agent Script specifications used to configure an Agentforce agent supply the structure for training environments and resolution checks.
That advantage comes with a clear limit in the paper’s comparisons. Koa remains below the strongest proprietary models on the broader benchmark table. Claude Opus 4.8 posts a 74.00 weighted Tau2Bench average and 78.18% on BFCL, while OpenAI’s GPT-5.5 reaches 83.99 and 67.63%, respectively. The models lead on different tests, but neither comparison supports a claim that Koa matches the frontier across the board.
The authors also caution against generalizing one of their training conclusions. In a scoped comparison, reinforcement learning produced substantially better multi-turn tool-use results than supervised fine-tuning, while supervised fine-tuning remained competitive on single-turn CRM tasks. Because the starting model had already received reinforcement-learning post-training, the paper says the finding may not apply to a foundation model before reinforcement learning.
A model aimed at the operational middle
Koa targets a gap between a broad foundation model and a fully managed enterprise agent system. Tool use in that setting requires more than producing a plausible response: the model must identify the right operation, supply valid arguments, respect which tools are available to a particular subagent and continue through several turns until the requested state is reached.
Salesforce’s results suggest that workflow-specific training can improve those behaviors without using customer records during training. They do not show that Koa is the best general-purpose model, nor do they establish how the model performs on live Salesforce deployments. The paper instead offers a narrower result: an open-weight model can gain measurable tool-use and CRM capability when reinforcement learning is driven by executable workflow specifications and task-resolution rewards.