The Pulse
Forcebench Finds Coding Agents Add 2–4 Points on Salesforce Tasks
Forcebench tested three locally served open-weight models on 60 Salesforce coding tasks and found that agent workflows raised pass rates by 2–4 percentage points, with no clear difference. The study also found that Salesforce skills were ra
AI.info Team · Written with AI ·
Three models gained no more than four points
Moving three open-weight models into a coding-agent workflow raised their scores by just 2 to 4 percentage points on a set of 60 Salesforce coding tasks, according to a Forcebench study. None of the gains amounted to a clear difference in the study’s paired comparisons. The result offers a narrow test of whether an agent’s file access, command execution and repeated work turns improve coding performance when the model already receives the task and relevant files.
Forcebench compared a single response with runs through OpenCode 2.0.21, using DeepSeek V4.1 Flash, Gemma 4 26B-A4B and Qwen3.8 27B. The models ran locally on Forcebench hardware, and each task used the same grading rules and a 32,768-token output budget. Scores measured pass@1: whether the first answer solved the task, without retries.
DeepSeek scored 60% as a single response and 62% in the agent. Gemma rose from 23% to 26%, while Qwen moved from 14% to 18%. The study’s confidence intervals leave substantial room for effects the test could not detect: Forcebench says the data cannot rule out changes of up to 13 points for the agent comparison.
The agent could not test against a Salesforce org
The experiment did not give OpenCode access to a Salesforce organization or Salesforce tools. The agent could read and edit task files and run shell commands, but it could not deploy code, run Apex tests or query Salesforce data. That distinction limits the conclusion: the test measures an agent working with files, not an integrated workflow that can check its changes against a live environment.
Forcebench says the tasks supplied the requirements, files and expected answer format in one message. In that setup, the agent’s added loop of trying, checking and fixing had little to verify beyond its own work. The study also tested only one agent and three locally served models, not hosted API models or other coding-agent products.
Salesforce skills were rarely loaded
Forcebench also tested 16 Salesforce skills covering areas such as Apex, Lightning Web Components, Flow, SOQL, permission sets and API references. The agent could see skill names and descriptions, then choose whether to load them. It rarely did: Gemma used a skill in 1 of 132 answers, Qwen in 20 of 132, and DeepSeek on 15 of its 60 tasks.
When the skills were supplied automatically on tasks they matched, results varied by model. DeepSeek’s score fell 21 points against its agent run without skills, while Gemma’s rose 16 points; Qwen’s increased 3 points, a difference the study did not find clear. The corresponding 95% intervals were −37 to −4 points for DeepSeek, +3 to +29 for Gemma, and −6 to +13 for Qwen.
Those skill results come from a smaller comparison across 24 tasks in six Salesforce-related suites. Gemma and Qwen each answered those tasks four times per condition, while DeepSeek answered once per task. Forcebench cautions that only two of nine comparisons on the page were clear on their own, so some apparent effects could be chance; its results do not establish that Salesforce skills generally hurt or help coding agents.
More turns also meant more time and input
Agent runs used more output tokens and took longer than a single answer, though Forcebench says those times cannot be compared directly because the single-response and agent runs occurred under different conditions. In the agent, DeepSeek averaged 661 seconds per task without skills and 761 seconds with them; Gemma averaged 75 and 85 seconds, and Qwen 86 and 93 seconds.
Skill descriptions added input to every request, and DeepSeek processed about twice as many input tokens with them as without. A few sessions ended without an answer because the model spent its first turn’s output budget thinking before it read a file. The next concrete test is whether these patterns change when agents can deploy, run tests and query an actual Salesforce org—the workflow this study’s setup could not evaluate.