The Pulse
TasteVal Finds Opus 5.5 Uses Half the Compute of Expert Researchers
P-Zero Research’s TasteVal benchmark finds that Opus 5.5 reaches expert-level results on eight AI research tasks with a 2.3x compute multiplier. The study also finds that the tasks test experiment design and interpretation—not which researc
AI.info Team · Written with AI ·
Opus 5.5 matched the benchmark’s expert baseline using about half the serial experimental compute, according to TasteVal, a new study of AI systems’ ability to design experiments and interpret results. The model recorded a 2.3x compute multiplier, with a 95% confidence interval of 1.15x to 4.37x. The result comes from eight AI research tasks built by P-Zero Research, not from a broad test of scientific discovery.
Eight tasks, one H100 at a time
Authors Oliver Jaffe and Dane Sherburn designed TasteVal to separate research judgment from coding. The evaluated model proposes experiments and reads their results; a fixed coding agent, Opus 4.8, implements and runs them. Each attempt can use up to 40 H100 GPU-hours or 120 hours of wall-clock time, whichever comes first.
The tasks cover areas including training language models, selecting pretraining data, modeling human preferences and testing resistance to adversarial prompts. Researchers recruited 24 human experts, with at least two assigned to each task, and used the strongest human attempt on each task as the comparison baseline. They evaluated 20 models released from 2023 through 2026, using six runs per model per task.
“The best-performing model, Opus 5.5, exceeds our expert baseline, with a compute multiplier of 2.3x (95% CI 1.15-4.37), at roughly 1/30 of our baseliners' average per-run cost,” Jaffe and Sherburn write in the paper.
Compute efficiency rose faster than final scores
The headline multiplier measures how much serial GPU compute a system needs to reach a score achieved by the expert baseline. A 2x multiplier means reaching the same score in half the compute on the same hardware; it does not mean the model uses half as many GPUs or produces twice the final score.
The authors estimate that frontier models’ compute multiplier doubled about every three months after December 2025, compared with roughly every 14 months before then. The estimate has a 95% confidence interval of 1.7 to 5.0 months. They found no corresponding jump in final normalized performance: that measure doubled every 14.6 months across the period studied.
The researchers say they have more confidence in the overall trend than in any single model’s multiplier. Performance varied substantially across tasks, and the analysis’s uncertainty is driven largely by the limited number of benchmark tasks. Opus 5.5’s result is therefore evidence about this test, not a general measurement of how quickly AI can conduct research.
What the benchmark leaves out
TasteVal tests what to try after a research problem has already been chosen. It does not measure whether a model can identify which problems are worth pursuing. Its tasks also provide clean validation signals and quick, relatively low-noise scoring on one H100—conditions that differ from many frontier research projects.
The paper’s review of experiment records found a further distinction between models and the strongest human runs: people more often measured variation across repeated runs and rejected improvements smaller than that variation. The authors also report that model experiments tended to make incremental changes rather than introduce new ideas. Their conclusion says progress so far has come from “shallow and moderate optimization rather than novel ideas.”
The benchmark’s tasks remain private so models cannot train on them before evaluation. TasteVal thus offers a repeatable way to compare experiment-selection efficiency as systems change, but its score cannot settle whether those systems can independently set a valuable research agenda. How much of the measured advantage would survive on slower, messier projects that lack a quick scoring signal?