The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks

AI summary

Overview

Research area: Evaluation and training of LLM agents on long-horizon tasks, specifically the measurement of decision quality ("taste"), plus knowledge distillation of that judgment into a smaller model.

Technical level: Intermediate. The paper is written in accessible prose, but assumes familiarity with agent benchmarks, two-choice evaluation protocols, position bias, and distillation methods such as token-level KL objectives and LoRA adapters.

One-sentence scope: The paper formalizes an agent's "taste" as its ability to pick the better of two directions at a mid-trajectory decision point before the outcome is visible, builds a 502-question benchmark mined automatically from real engineering and research trajectories, shows that frontier models score poorly, and demonstrates that this judgment can be distilled into a student model that improves end-to-end task success.

What This Paper Is About

LLM agents increasingly run long tasks where early decisions — which hypothesis to test, which implementation to build on — determine the final outcome, but existing benchmarks only report whether the agent finished the task, not whether the decisions along the way were good. Judging those decisions directly is hard because a good choice and a bad choice look equally reasonable at the moment of choosing, and expert human annotation is expensive and does not scale to new domains. This paper shows that the later part of a trajectory provides hindsight evidence for earlier decisions, uses that insight to build an automatically labeled benchmark called Taste-Bench, and then shows that the judgment can be trained.

Key Contributions

1. A formalization of agent taste. The paper defines taste as the ability to choose the better direction at a decision fork — a point where parallel attempts or a self-correcting detour show that one direction leads to a better outcome — and shows this ability can be measured from hindsight over already-recorded trajectories without human annotation.
2. Taste-Bench, a released benchmark of 502 taste questions. Questions cover software engineering (390) and machine-learning/AI R&D research (112), are mined in a 2 × 2 design crossing construction type (parallel or detour) with domain, and are publicly released with code.
3. Evidence that questions get harder with time horizon. Forks are annotated with four ordinal time-horizon levels, and accuracy falls as the horizon lengthens; a larger reasoning budget does not help.
4. A demonstration that taste is trainable. Distilling the reasoning of a teacher that has seen the supported candidate into a student (base model Qwen3.6-27B with LoRA adapters) improves judgment on unseen tasks and improves end-to-end success when the student's judgment is injected as advice.

Main Findings

• Frontier models show limited taste. On the 502 questions, the best model, GPT-5.6 Sol, reaches an Average accuracy of 59.7%, with GPT-5.5 close behind at 59.5%, while the other models are widely dispersed below them. Because a question counts as correct only when both candidate orders are answered correctly, random guessing scores 25% and always picking the same position scores 0%.
• Detour forks are harder than parallel forks. Across the four cells of the release, detour forks are harder than parallel forks in both domains, and this gap exceeds the gap between the two domains.
• Accuracy collapses as the time horizon grows. Averaged over the 14 evaluated models, accuracy falls from 62.3% at the "in prefix" level to 21.0% at the "more work" level, which is near the 25% random-guessing score.
• More reasoning does not help. Rerunning under three reasoning-effort settings for two models (questions, prompt, token limit, and protocol unchanged; 6,024 responses in total) moved GPT-5.6 Sol's accuracy by −0.2 points and GPT-5.6 Luna's by +2.2 points. The models spend the most reasoning tokens at the "more work" level, which is also their lowest-accuracy level.
• Taste is not a restatement of end-to-end ability. Compared with public SWE-bench Verified scores on the Vals AI leaderboard (11 models after excluding three whose responses were unparsable on more than 9% of presentations), the Pearson correlation with Average accuracy is r = +0.63, so SWE-bench Verified explains R² = 0.39 of the variance. On the engineering subset alone the correlation is only r = +0.37. The four highest models on SWE-bench Verified sit within 4.0 points of each other there but are 10.7 points apart on Taste-Bench Averages.
• Mined labels agree with human review. Reviewing 100 sampled questions with two reviewers produced 172 explicit A/B judgments, of which 170 agreed with the mined label, giving 98.8% agreement. On the 74 questions where both reviewers chose A or B, they agreed with each other 98.6% of the time, with Cohen's κ = 0.973.
• Taste transfers to unseen tasks. On the held-out, task-disjoint fold of the 390 engineering questions, the distilled student reaches 47.9% accuracy versus 30.0% for the base model under the two-order protocol, and mean accuracy over the two orders rises from 42.7% to 62.4% — a gain of 17.9 percentage points on tasks never seen in training. (On the training fold, the fraction answered correctly in a single order rises from 48.6% to 92.9%, which the paper notes cannot by itself distinguish learning from memorization; GPT-5.6 Sol scores 56.9% on those held-out engineering questions.)
• Better judgment improves end-to-end success. On 41 held-out SWE-bench Pro tasks, a fixed Qwen3.6-27B executor scores 14.6% with no advice, 39.0% with correct advice at every fork (a 24.4 percentage point gain, described as the upper bound), and 33.7% with student advice (a 19.1 percentage point gain).

Methodology in Plain English

The central trick is to use a trajectory's own ending as a label for its middle. The researchers collected two pools of real agent work: 2,677 graded rollouts produced by GPT-5.4 and GPT-5.5 agents on 517 SWE-bench Pro tasks across 11 repositories (31 runs between April and July 2026), and 1,132 agent runs on 47 AI R&D tasks from RE-Bench and the research subset of HCAST, downloaded from METR's public transcript release MALT.

From these pools, a generator model (GPT-5.6 Sol at high reasoning effort) proposes candidate forks in two ways. In the parallel construction, two attempts at the same task share an equivalent prefix and then diverge, and the recorded outcome of each branch says which direction was supported; the construction starts from 600 pairs of attempts with opposite outcomes, drawn from 905 candidate pairs. In the detour construction, a single trajectory shows the agent committing to a direction, hitting an observed failure, and recovering with a different direction that completes the task; the fork is placed at the step just before the abandoned direction, and the generator must cite four verbatim excerpts in order (896 trajectories in a first pass and 437 in a second pass supply these).

Each candidate then passes a rubric with mechanical checks, and surviving questions are filtered two ways: a question is discarded as trivial if every judge answers it correctly using only the two candidate descriptions without the trajectory, and discarded as undecidable if any judge reading the full record, trajectory, and outcome disagrees with the label. Of 4,657 candidate forks the generator proposed, 10.8% passed all filters, yielding 502 questions.

For evaluation, each question is presented twice — once in a deterministic seeded order and once reversed — to counter position bias, and a question counts as correct only if both orders are answered correctly. Fork difficulty is annotated with a four-level time horizon (in prefix, inferable, next step, more work) depending on which part of the record justifies the supported candidate.

For training, the paper fine-tunes a Qwen3.6-27B base model with LoRA adapters on two task-disjoint folds of the 390 engineering questions, so no student sees the source task of an evaluation question. Instead of fitting the binary labels, it distills the free-form reasoning of a privileged teacher (the same frozen base model, given a short demonstration of the supported candidate) into a student context containing only what the benchmark shows — the task, the prefix, and the two shuffled candidates — using an SDPO-style token-level distillation with a forward KL over sampled continuations. At deployment, the student's choice at each fork is written as an "advice" note (the situation, the candidate to avoid, the candidate to take) into the executor's task context, and a fixed Qwen3.6-27B executor completes the task independently.

Why This Matters

Impact on research. The paper argues that end-to-end pass rates hide the quality of the decisions made along the way, and that process-level judgment for long-horizon agents has been measurable only through expensive expert annotation. By mining hindsight labels from trajectories agents already produce, the approach scales with the volume of agent work rather than with annotation budgets, and it offers a diagnostic that is only partly correlated with existing agent benchmarks.

Real-world applications (from the settings the paper studies and measures):

• Software engineering agents selecting which repository change or implementation approach to build on across long multi-step fixes.
• Machine-learning and AI R&D agents choosing which hypothesis or experiment to pursue next.
• "Advisor" style systems in which a small trained model steers a separate, larger executor with per-fork advice.
• Reusing existing agent logs as a training signal instead of discarding them after each run.

Industry relevance. The distillation result shows that a small student model (Qwen3.6-27B with LoRA adapters) can supply advice that lifts a fixed executor's SWE-bench Pro success rate from 14.6% to 33.7% on tasks it never trained on, reaching most of the 39.0% upper bound set by fully correct advice. That points to a practical pattern — mine forks from production agent traffic, train a judgment model, and inject its advice into task context — rather than relying on larger reasoning budgets, which the paper finds do not move taste accuracy.

Future Directions

• Understanding why reasoning budget fails. Accuracy is unchanged across three reasoning-effort settings, and models reason longest where they are least accurate. Whether a different mechanism — retrieval, tool use, or explicit simulation of future steps — could help is left open.
• Closing the gap between student advice and correct advice. The executor reaches 33.7% with student advice against 39.0% with correct advice, so part of the available gain is unrealized. Improving the student, or combining its judgments differently, is an obvious next step.
• Extending beyond the two studied domains. The release covers engineering and research, and the paper notes that human annotation is hard to scale across new domains; whether the mining pipeline generalizes to other long-horizon domains is not established.
• Handling long time horizons specifically. Because accuracy falls to 21.0% at the "more work" level, the hardest cases — where justification requires completing a local check or substantial later work — remain largely unsolved by the current distillation recipe.

Target Audience

Researchers and engineers working on LLM agents for software engineering and scientific/ML research; benchmark designers interested in process-level or step-level evaluation; practitioners building agent training pipelines from logged trajectories; and readers interested in knowledge distillation and advisor-style architectures. Some familiarity with agent evaluation protocols and fine-tuning is helpful, though the main text is written to be readable without deep background.

Authors’ abstract

LLM agents increasingly work on long-horizon tasks, and the decisions they make along the way, such as which hypothesis to test or which implementation to build on, determine the outcome of the whole run. Making these decisions well is becoming a key capability for both engineering and research agents. We refer to the ability to make good long-horizon decisions as the taste of an agent. While existing benchmarks measure the end-to-end success of agents on long-horizon tasks, none of them measures the taste of an agent. To address this problem, we build Taste-Bench, a benchmark of taste questions constructed automatically from trajectories that agents produced in engineering and research tasks. Each question presents a decision fork, a point in a trajectory where multiple directions are available and one of them leads to a better outcome, and the evaluated model chooses among these directions without seeing what happens after the fork. We mine these forks automatically from parallel attempts at the same task and from detours inside a single trajectory, without needing human annotation. We evaluate frontier models on Taste-Bench and find that the best model answers only 59.7% of the questions correctly. We further find that forks whose deciding evidence appears later in the trajectory are much harder for every model, and that a larger reasoning budget does not improve the accuracy. Finally, we show that taste can be trained. We distill the judgment of a teacher that has seen the outcome into a student model, and the student makes better decisions on unseen tasks and improves end-to-end success on held-out SWE-bench Pro tasks.