The Pulse
Taste-Bench Finds a Top Score of 59.7% Among 14 Models
A paper introduces Taste-Bench, a benchmark that tests whether AI models can choose a promising direction before seeing how it turns out. The highest-scoring model reached 59.7%, while performance fell to 21.0% on choices requiring more lat
AI.info Team · Written with AI · fact-checked ·
“We evaluate frontier models on Taste-Bench and find that the best model answers only 59.7% of the questions correctly.”
Wenbo Pan and co-authors, authors of the paper, write
Wenbo Pan and eight co-authors posted their paper to arXiv on September 22, 2026, introducing Taste-Bench, a test of whether AI models can choose the better direction before a task’s outcome is known. The paper was revised the following day. Its central result: the strongest of 14 evaluated models answered fewer than six in 10 decision questions correctly.
The researchers call this ability an agent’s “taste.” Their benchmark targets choices made during extended engineering and research work—such as which software implementation to build on or which experiment to pursue—rather than simply checking whether an agent eventually completes a task.
Taste-Bench tests the choice, not the finish
Each of the 502 questions presents a task, the recorded steps leading up to a decision point, and two possible next moves. The model must select one without seeing the later work that reveals which direction produced the better outcome. The questions come from software engineering and machine-learning research trajectories.
The team built questions in two ways. “Parallel” forks compare separate attempts at the same task that diverged and recorded different outcomes. “Detour” forks capture an agent abandoning a direction after a failure and recovering by taking another route. The latter tests whether a model can spot a poor move before the agent in the recorded run did.
The authors’ pipeline proposed 4,657 forks, but only 502 survived filters meant to remove questions that could be guessed from the options alone or whose outcomes did not clearly support a label. Of the released questions, 390 concern engineering and 112 concern research. In a review of 100 sampled questions, human reviewers’ explicit choices agreed with the mined labels 170 times out of 172.
Later evidence pushes scores toward chance
GPT-5.6 Sol scored 59.7% on the benchmark’s average measure, which gives equal weight to the engineering and research subsets. GPT-5.5 followed at 59.5%. The score is not a percentage chance of finishing a task: it measures choices across benchmark questions, and a question counts as correct only if the model selects the right option in both versions, with the options presented in opposite orders.
That scoring rule sets random guessing at 25%. Performance fell as the evidence needed to judge a choice appeared later: averaged across the 14 models, accuracy dropped from 62.3% when the decisive information was already in the visible task history to 21.0% when it required more work. The latter result sits below the benchmark’s random-guessing reference.
Giving models more reasoning effort did little to close the gap in the paper’s limited test of two models. Moving from the lowest to the highest setting shifted GPT-5.6 Sol’s accuracy down 0.2 percentage points and GPT-5.6 Luna’s up 2.2 points. The study therefore finds no meaningful improvement from that change in those two tests, not proof that reasoning effort never helps other models.
Distillation lifts an engineering model’s score
The authors also tested whether training could improve the measured judgment. They distilled a teacher’s outcome-informed reasoning into Qwen3.6-27B, then evaluated the student on engineering questions from tasks excluded during training. Its accuracy rose from 30.0% for the base model to 47.9% for the student.
In a separate test on 41 held-out SWE-bench Pro tasks, a fixed executor completed 14.6% of tasks without advice and 33.7% with advice from the distilled student. Supplying correct advice at every decision point raised success to 39.0%, the authors’ upper bound for that setup. The advice was mined from earlier runs on those same tasks, so the result tests whether it helps a new run—not whether the student alone accounts for all of the improvement.
Recorded outcomes are useful, but imperfect evidence
Taste-Bench uses what happened later in recorded trajectories to label earlier choices. That makes it possible to build questions without asking experts to annotate every decision, but an outcome can also reflect execution quality and environmental factors. The authors say they filter for cases where the recorded outcome clearly follows from the direction chosen; the human review offers a check on those labels, not a randomized test of each decision’s effect.
The benchmark’s contribution is a separate measure of agent behavior: not whether a system reaches the finish, but whether it picks a supported path while the consequences remain hidden. Its strongest model scored 59.7% under the paper’s paired-order test, and its hardest category—choices justified only by more work—averaged 21.0% across the evaluated models.