The Pulse
Vals AI Finds Agent Teams Cost Up to 5.1 Times More
Vals AI tested GPT 6 Sol and Claude Opus 5.5 as single agents and agent teams on 50 web-app builds. Teams cost up to 5.1 times more, but only one of four score comparisons produced a statistically significant gain.
AI.info Team · Written with AI ·
“Teams cost 1.8 to 5.1 times as much as single agents, and only for Sol at medium effort was the difference in score statistically significant.”
Katrina Drozdov, author of the Vals AI post
Vals AI’s October 9 study found that splitting software work among AI agents often raised costs without reliably improving results. Across tests of OpenAI’s GPT 6 Sol and Anthropic’s Claude Opus 5.5, only one of four team-versus-single-agent score comparisons showed a statistically significant gain.
The study used Vals AI’s Vibe Code Bench, where agents build complete web apps from product specifications. Browser agents graded the finished apps by running their user-interface tests. The work could be divided into database, API, interface and deployment tasks, giving teams opportunities to build parts in parallel.
One significant gain in four comparisons
Vals AI tested 50 apps at two reasoning-effort settings, medium and maximum. In each setup, the team lead received the same instruction to divide the work, delegate independent pieces, then integrate and verify the result. Each of the eight configurations ran once, so the statistical tests measure variation across apps, not variation across repeated attempts.
GPT 6 Sol’s medium-effort team passed 84.9% of UI tests, compared with 77.6% for its single agent, a gain of 7.3 percentage points. Vals AI reported that difference as statistically significant, with p = 0.005. The team cost an average of $3.07 per app, against $1.22 for the single agent, and finished in a median 11.9 minutes—about half a minute faster.
The other score changes were not statistically significant: Sol’s max-effort team gained 1.4 points, Opus’s medium-effort team lost 0.3 points, and Opus’s max-effort team gained 3.4 points. Opus’s max-effort team recorded the highest score, 93.2%, but its single-agent counterpart scored 89.8%; the study says that 3.4-point difference was not significant.
Opus’s maximum-effort team cost $122 per app
Team costs ranged from 1.8 to 5.1 times the comparable single-agent costs. For Opus at maximum effort, the team averaged $122 per app, while the single agent averaged $23.77. The team also took a median 174.1 minutes to finish, compared with 74.7 minutes for the single agent.
Vals AI traced much of the extra expense to repeated input. Each subagent had its own working context, including the app specification, which was sent again with later model calls. Opus’s max-effort subagents read a median 224 million cached tokens per app, the post says, compared with 55 million for the single agent. One team run hit the time limit while verification was still underway; Vals AI estimated its cost at about $189 from token counts and included that estimate in the team’s average.
The cost figures are estimates based on list prices: Claude Code reported the Opus totals, while Sol’s totals used OpenAI base-tier prices and reported token use. Vals AI says it could not evaluate higher charges for requests above the 272,000-token long-context threshold because usage was reported per run rather than per request.
Parallel work helped Sol more than Opus
The two systems also organized teams differently. Sol’s lead generally delegated early and tested the assembled app itself; at maximum effort, its team’s median runtime fell to 27.6 minutes from 38.4 minutes for the single agent. Opus’s lead usually drafted a shared contract describing the app’s schema, API routes and file ownership, then delegated in successive waves of builders and testers.
That sequence brought a time penalty. Opus’s maximum-effort team spent a median 67.8 agent-minutes testing after its first deployment, compared with 9.6 for the single agent. Across its team runs, the median completion time was more than twice as long as the single-agent time.
The result applies to one benchmark
Vals AI cautions that the findings come from one set of full-stack web-app tasks and may not hold for other work. The comparison also combines subagents with a delegation prompt, rather than isolating the effect of subagents alone. And because Sol and Opus ran through different software harnesses, the study cannot cleanly attribute differences between the models themselves.
For teams considering agent systems on similar app-building jobs, the measured trade-off is specific: Sol’s medium-effort team added 7.3 points for an extra $1.85 per app and a slightly shorter run. Opus’s maximum-effort team cost $98.23 more per app than its single-agent counterpart, took about 99 minutes longer at the median and did not produce a statistically significant score gain.