rTaste: Do Research Agents Have Taste? Evaluating Early Stage Research Judgment

AI summary

Overview

Research area: Natural Language Processing / evaluation of AI research agents (LLM agents that propose and run experiments), sitting at the intersection of agent benchmarking, forecasting calibration, and the study of research judgment or "taste."

Technical level: Intermediate. The study design is easy to follow, but the metrics used (Brier scores, confidence intervals across seed papers, Cohen's kappa for auditor agreement) assume some familiarity with empirical evaluation practice.

Scope (one sentence): The paper measures whether six frontier models, acting as research agents alone or as teams, can choose and forecast small experiments from a recent NLP paper well enough to beat trivial baselines.

What This Paper Is About

The step of research that has not been handed to AI agents is "taste": judging which experiments are worth running before anyone knows how they will turn out. The paper tests this by giving six frontier models seven recent NLP papers and asking each to propose five minimum viable experiments (MVEs), forecast whether each will clear a threshold the model itself sets, and rank them. Crucially, every proposed experiment is then actually executed, including the ones ranked last, so the agents' expectations can be checked against real outcomes.

Key Contributions

1. An operational definition of research "taste" that can be scored. An MVE counts as useful only if its outcome is not obvious in advance and the agent's expectation about it is accurate enough to act on; the paper turns this into registered thresholds, probabilities, rankings, and audited outcomes.
2. An evaluation harness with built-in baselines and controls. Two harnesses (a single researcher working in one context, and a research team of two proposers, a critic and a lead) are run for every model and every seed paper, alongside a fresh "same model judge" that sees only the paper and the five registrations, and a random-pick baseline for rankings.
3. A released dataset of 415 MVEs with their forecasts, executor implementations, rerun values, audited outcomes, novelty labels, and post-hoc reviews, plus the harness, at https://github.com/ArtKanke/rTaste.
4. Empirical answers on quality, forecasting, ranking, and cost. The paper reports how many MVEs are judged sound, valuable and new; whether forecasts beat a base rate and a same-model judge; whether first choices beat random picking and the experiment written first; and what teams cost relative to what they add.

Main Findings

• Most experiments ran and most were decent, but few went beyond the paper. Of 415 MVEs, 194 were judged sound, valuable and new, and only 73 of those were also judged beyond the seed paper and non obvious. Nearly all MVEs were delivered (412 of 415) and most were judged sound.
• Model size separates quantity of good MVEs. The four larger models (GPT-6 Astra, Claude Fable 5.1, Claude Opus 5.5, GPT-6.1 Sol) propose about three sound, valuable and new MVEs in five (single-researcher means of 2.57, 2.43, 3.33 and 3.00); the two smaller models, GPT-6 Luna and Muse Spark 1.3, about one (0.71 and 0.14 for single researchers, 2.00 and 1.00 for teams).
• Two models account for most of the non-obvious work. GPT-6 Astra and GPT-6.1 Sol proposed two thirds of the MVEs rated beyond the paper and non obvious; within seed papers their share exceeds that of Claude Fable 5.1 and Claude Opus 5.5 by 48 percentage points (95% CI 29 to 66).
• Smaller models' experiments often leave the question open. GPT-6 Luna's MVEs are sound as often as GPT-6.1 Sol's, but reviewers read the evidence as informative about half as often (38% against 74%, as stated in the text).
• Agents mostly build on the paper. By reviewers' answers, 66% of delivered MVEs build on the seed paper, 24% check one of its claims, and 5% open a direction the paper merely suggests; 44% of all MVEs match an experiment on a model's own list of obvious follow-ups.
• Agents converge on the same experiments. On one seed paper (Nguyen-Tri et al., 2026), all eight conditions of the four larger models wrote the same check of the appendix correctness argument first.
• Stricter experiment design is a model signature. GPT-6 Astra and GPT-6.1 Sol often compare a confidence-interval bound with the threshold rather than a point estimate (59 and 63 MVEs respectively), and their registered events occur in 25% of their sound MVEs, versus 52% for Claude Fable 5.1 and Claude Opus 5.5.
• A non-event is often still informative. Reviewers judged the evidence informative for 50% of sound MVEs whose registered event did not occur, mostly as evidence against the hypothesis; they judged it inconclusive for 20% of sound MVEs whose event did occur.
• Forecasts beat a base rate but not an outside reader. The agents' Brier score is 0.192 against a base rate of 0.240, but the difference from the same model judge is −0.006 (95% CI −0.021 to 0.009), and the mean of the six judges' probabilities is more accurate at 0.179.
• Overconfidence runs in both directions. GPT-6 Astra, GPT-6.1 Sol, GPT-6 Luna and Muse Spark 1.3 state higher probabilities than the rates at which their events occur, while Claude Fable 5.1 and Claude Opus 5.5 state lower ones.
• First choices beat random but not the experiment written first. The agent's first choice is the auditors' most valued MVE in 38% of conditions (random pick: 20%), but its advantage over the MVE written first is 3 percentage points (95% CI −8 to 15), and the first choice is sound, valuable and new no more often than a random pick.
• Rankings track writing order and likely passes. The first choice is the MVE written first in 47% of conditions; the MVE written first is on an obvious follow-up list in 61% of conditions against 35% for the fifth; the first choice's event occurs in 47% of conditions versus 29% for the last choice.
• Teams cost more without a demonstrated quality gain. Teams produce 0.71 more sound, valuable and new MVEs of five on average, but the 95% CI spans zero (−0.10 to 1.52). A condition cost $15.31 for a single researcher and $23.93 for a team; per $100, single researchers produced 13 such MVEs and teams 11.
• Cost of proposing varies far more than cost of running. Proposing cost differs about a hundredfold between models (GPT-6 Luna to Claude Fable 5.1), while implementing and rerunning an MVE differs about twofold because the same two executors served all models.
• Executor choice affects audit outcomes. Auditors judged 18% of MVEs implemented by the Claude Opus 5.5 executor not sound, compared with 3% for the GPT-6.1 Sol executor, while evidence was informative equally often under both.

Methodology in Plain English

The researchers took seven NLP papers posted to arXiv between 3 August and 24 September 2026 that had released code and could be probed on a single GPU without models larger than 8B parameters; papers this recent were chosen so that little follow-up work existed for an agent to find and repeat. Each of six models was given the paper's full text, page images and read access to its code repository, with no suggested research gap or hypothesis, and worked once as a single researcher and once as a research team (two independent proposers, a critic, and a lead). The model had to write five registrations, each specifying a hypothesis, the closest prior work, and one "registered event": a statistic, a relation and a threshold fixed in advance. Before anything ran, the agent gave each MVE a probability and a rank, knowing all five would run.

Three comparison layers were added. A fresh instance of the same model acted as a judge, seeing only the paper and the five registrations, and produced the same forecasts and ranking. Two novelty judges searched arXiv to label each MVE as equivalent, a close variant, or distinct, and each of the six models also wrote ten obvious follow-up experiments from the paper alone so that obviousness could be checked. Then two executors (Claude Opus 5.5 and GPT-6.1 Sol, alternating across seed papers) each implemented one MVE per context without seeing the ranking, forecasts, or other MVEs, on an A100 GPU with a 45-minute run limit and no hosted-model access. Results were deleted and each evaluation was rerun on a machine the executor never used to determine whether the registered event occurred. Three auditors (Claude Opus 5.5, GPT-6.1 Sol, Grok 4.7) judged faithfulness, validity, and research value from 0 to 3 without seeing the seed paper or who proposed the work, taking majority and median votes. Finally, three reviewers re-read every delivered MVE to say what kind of experiment it was, how obvious it would be to ten researchers, and what the evidence said about the hypothesis. Comparisons are made within seed papers and averaged over the seven, with 95% confidence intervals computed across them.

Why This Matters

Impact on research. The paper separates two capabilities that research-agent benchmarks tend to blur: checking and extending a paper (which current agents do reasonably well, and which can be done for a few dollars per experiment) and proposing genuinely new directions (which they mostly do not). It also shows that a model's own reading and search added no measurable forecasting advantage over a reader who saw only the registrations, which matters for anyone planning to let agents allocate research budget.

Real-world applications.

• Automated paper triage: running cheap controlled checks of a new paper's claims before a lab invests in a full replication.
• Research portfolio screening: using registered thresholds and pre-execution probabilities as a lightweight filter for which internal directions get funding.
• Agent procurement and routing: picking models by the kind of experiment they propose, since the larger models differ less in soundness than in whether they push beyond the source paper.
• Cost planning for agentic research pipelines: the reported per-condition costs ($15.31 for a single researcher, $23.93 for a team) and the hundredfold spread in proposal cost give a concrete basis for budgeting.

Industry relevance. Frontier labs report that agents write much of their code and support a growing number of experiments, yet a risk assessment inside four frontier labs found no evidence that any relied on agents to set research agendas or make budget decisions. This paper quantifies why that boundary may still be sensible, and it releases a harness that labs could reuse to track whether it moves.

Future Directions

• Turn rankings into a real budget decision. Give an agent a fixed research budget to allocate across its own MVEs and to revise after the first results, then ask whether the money goes to experiments that turn out sound, valuable and non obvious.
• Add a human baseline. Every label in this study comes from models; running the same protocol with human researchers would supply the missing comparison for both proposals and budget allocations.
• Test rankings that actually decide what runs. Because every MVE here ran regardless of rank, an agent that knows its ranking controls execution might reason about it differently.
• Measure stability and audit quality. Each condition ran once, and auditor agreement was poor on faithfulness (pairwise Cohen's kappa of 0.32, 0.30 and 0.09), so repeat runs and better-grounded judging are open problems.

Target Audience

Researchers and engineers building or evaluating LLM research agents; benchmark designers who need baselines that separate execution from judgment; science-of-AI and meta-research scholars studying how automated systems choose problems; and research-ops or lab leadership deciding how much agenda-setting to delegate to agents. Readers interested in forecasting calibration and in multi-agent versus single-agent designs will also find directly relevant evidence.

Authors’ abstract

Do research agents have taste, the judgment of which experiments are worth running before their outcomes are known? We give six frontier models seven recent NLP papers. Each model, working as a research agent alone or as a team, proposes minimum viable experiments (MVEs), forecasts whether each will pass a threshold it sets and ranks them. A separate executor then attempts every MVE, including those ranked last, and three model auditors judge the results. Of 415 MVEs, 194 are judged sound, valuable and new, but fewer than half of these (73) are also judged beyond the seed paper and non obvious. The agents' first choices match the auditors' most valued experiment more often than a random pick, but are not shown to improve on choosing the experiment written first. Their forecasts beat a base rate but are not measurably more accurate than those of a fresh instance of the same model given only the paper and registrations. Teams cost more without a clear overall improvement in proposal quality. The agents produce many worthwhile experiments, but the evidence that their rankings improve the choice of which to run remains limited. The evaluation harness and a dataset of all 415 MVEs, with their forecasts and audited outcomes, are available at https://github.com/ArtKanke/rTaste.