The Pulse
Vals AI Finds Four Models Kept at Least 80% of Their Scores at Lower Effort
At their lowest reasoning setting, four of seven models kept at least 80% of their Vals Index score for half the cost or less. The study also found that lower settings sometimes matched leaderboard scores at lower cost, while results varied
AI.info Team · Written with AI · fact-checked ·
Claude Opus 5.5 kept 80% of its Vals Index score while using 16% of the cost at its lowest reasoning setting. Grok 4.7 kept 86% of its score at the same 16% cost. Across the seven models tested, four retained at least 80% of their score for half the cost or less. Vals AI published the results on November 3, 2026.
Lower effort cuts cost faster than score
The study compares each model’s lowest and middle reasoning settings with its leaderboard setting, usually its highest. Gemini’s leaderboard runs set no effort level, so Vals AI used the high setting for that comparison. At its lowest setting, DeepSeek V4.1 Flash kept 89% of its score at 50% of the cost, while GPT-6 Astra kept 88% at 18%. GLM-5.3 kept 81% at 53%, just over the study’s half-cost threshold.
Gemini 4 Argon behaved differently: its lowest setting kept 99% of its score but cost 84% of the leaderboard setting. At the middle setting, it scored slightly above its leaderboard score while costing 78% as much. Across the index as a whole, none of the seven models matched its leaderboard score at its lowest setting.
Lower settings sometimes match leaderboard scores
Across 56 model-and-benchmark comparisons, a lower setting scored as high as the leaderboard setting in 22 cases, at lower cost. Those 22 results were not significantly different from the leaderboard setting. Separately, among the cheaper settings tested overall, none scored significantly higher than the leaderboard setting, while 33 scored significantly lower. The 33 significantly lower results were among the cheaper settings tested overall, not only the 22 cases that scored as high as the leaderboard setting.
More reasoning made the largest median difference on Legal Research, where scores rose 19.2 points from the lowest to the highest setting. Code Migration 1.1 rose 15.6 points and Terminal-Bench 4 rose 12.1. By contrast, the median increases for Finance Agent 2.1, HLAB 1.1 and EMB 1.1 were each below six points.
Task-by-task selection is an upper bound
Vals AI also calculated what would happen if each task received the cheapest setting that achieved its highest score. With that hindsight, the median gain across models was 6.0 points over the leaderboard setting, while the median cost was 64% of leaderboard cost. Using the lowest setting everywhere cost 29% of the leaderboard setting, so the task-by-task selection spent more while scoring better.
The 6.0-point gain is an upper bound, not a result a system could count on in advance: the selection uses each task’s observed results, while a real system must choose its setting before seeing them. Vals AI ran the models on fixed subsets covering about half the tasks in each of the eight benchmarks, and most benchmarks had one run per setting. The company also notes that most leaderboard-setting runs came from earlier months than the other settings, so provider-side changes between runs could affect the comparisons.
The eight benchmarks were Finance Agent 2.1, EMB 1.1, Terminal-Bench 4, Vibe Code Bench, Code Migration 1.1, Legal Research, HLAB 1.1 and Tax Agent Bench. Vals AI says the subsets were selected so they would rank models in the same order as the full benchmarks. For most models, the leaderboard-setting results came from runs already on the leaderboard, rescored on the selected subsets; where there was no such run, the company ran that setting again.
The Vals Index combines agentic model performance across finance, coding, legal and tax tasks, weighted by each sector’s share of U.S. gross domestic product. Vals AI calculates its cost measure as the plain mean of the eight benchmark costs, estimating Code Migration cost from tokens. Cost per test uses list prices.
To assess score differences, the company used paired bootstrap tests, resampling the same tasks for both settings, and adjusted p-values across comparison groups. A difference counted as significant when its adjusted p-value was below 0.05. Vals AI cautions that no significant difference does not mean two scores are equal: the tests used about half the tasks per benchmark and could miss small differences. It also says the task-by-task calculation selects settings using the results from those same tasks, which makes the reported gain an upper bound.