IdeaTaste: Do LLMs Share the Research Taste of Human Researchers?

AI summary

Overview

Research area: Natural Language Processing / AI-for-science evaluation — specifically benchmarking the "research taste" of LLM-based research agents against real publication outcomes.

Technical level: Intermediate. Readers should be comfortable with LLM agent pipelines, retrieval over citation graphs, and benchmark metric design (matching, Jaccard overlap, weighted slot scoring).

Scope: IdeaTaste is a benchmark that freezes a research area's literature at the end of 2025, has populations of LLM agents write research proposals from that snapshot, and scores them against the papers the same human community actually published in 2026 — across four areas, six language models, four agent methods, and more than 90,000 proposals.

What This Paper Is About

AI scientists can now execute much of the research loop, but executing research well only matters if the research is worth doing — and that judgment, called research taste, is far less understood. Existing evaluations judge single ideas through expert ratings, fixed-target benchmarks, or model judges whose verdicts are unreliable, and population-level studies only measure how much models repeat themselves. This paper supplies the missing external reference: the papers a human community actually published after a literature cutoff, which record which directions its researchers chose. The goal is to measure how closely LLM agents' choices of what to read and what to propose resemble those of the human community working in the same area.

Key Contributions

1. A benchmark built on future publications. IdeaTaste freezes a literature corpus U through 2025 and uses papers published in 2026 as the human reference H, then scores LLM agent proposals at two levels — the literature they cite and the ideas they state — using four metrics: Match-Cite, Diversity-Cite, Match-Idea, and Diversity-Idea, each in [0, 1] and each read against the level the 2026 papers reach among themselves.

2. A large-scale population-level evaluation. Experiments span six contemporary language models and four methods (free, topic, landscape, frontier) across four areas from established to emerging, covering more than 90,000 proposals, with each agent given a 20-step budget and each population holding 300 proposals (frontier writes these in two rounds of 150).

3. Diagnosis of where the gap lies. Through direction coverage (task/method/pair tags), a slot-level decomposition of the idea summary (contribution type, problem, motivation, proposed approach, research plan), concrete case comparisons, and population-scaling experiments up to 3.8–17 times larger, the paper localizes the shortfall to turning a recognized problem into a concrete approach rather than to choosing a direction.

4. Blinded expert review plus a live leaderboard. Eleven experienced reviewers assess 192 documents without knowing their source (half from matched pairs, half with no counterpart, two reviews each), and the benchmark is released as an open, continuously updated leaderboard at https://cxcscmu.github.io/IdeaTaste that scores proposals written today against the communities' next publication cycle.

Main Findings

• LLM agents fall far short of community taste. On average, populations reach about 26% of the human level on Match-Cite and only 18% on Match-Idea, and no population under any model or in any area reaches half of the human level on either metric. For reference, the human level in machine learning systems is 20.84 Match-Cite and 13.56 Match-Idea; in code generation 18.75 and 11.25; in world models 16.38 and 13.87; in automated research 18.51 and 10.53.

• The gap is narrowest where the field is smallest. The shortfall is smallest in automated research, the newest and smallest area, and widens as the area grows, peaking in machine learning systems and code generation — LLM agents track a community most closely where it has least to choose from.

• They read widely but propose narrowly. For five of the six models, matching falls further behind on ideas than on citations. Once topic or frontier assigns a direction, four of the six models cite up to three times as diversely as the community, yet no population reaches the human level of idea diversity — citing the literature a community builds on is not enough to arrive at the ideas it pursues.

• More knowledge, from training or context, does not close the gap. The two models trained before 2026 match more human ideas — about 26% of the human level versus 14% for models that may have seen the reference. Across methods, matching stays within a narrow band while citation diversity varies roughly threefold, and the richest method, landscape, ranks last on Match-Cite in 19 of 24 settings.

• Agreement concentrates on problems, not approaches. Per-slot matching shows both Llama 4 Scout and DeepSeek-V4-Flash reach the problems and motivations of human papers many times more often than their proposed approaches and research plans; almost no human approach or research plan finds a counterpart among proposals.

• Directions are reached; combinations are not. In automated research with DeepSeek-V4-Flash, populations given a direction come close to the community on tasks alone and methods alone (frontier misses only a small share of each), but every population falls away on task–method pairs, where even frontier reaches only about half of the human tag mass.

• Concrete cases show the mechanism. Two human papers are paired with two proposals each from one model sharing the same problem: matching proposals modify the network itself (fine-tuning the router toward expert reuse; updating low-rank adapters online), while non-matching proposals leave the trained network unchanged and add machinery around it (prefetching experts around a fixed network; adapting through retrieved transitions).

• Larger populations do not help. Growing free and topic populations of Gemini 3.5 Flash-Lite and DeepSeek-V4-Flash from 300 proposals to the size of the human reference (3.8 to 17 times larger) yields curves that rise gently and level off well below the human level.

• Blind review: the gap is about execution, not writing. AI proposals trail 2026 papers by 20 points on expected success (50.39 vs 70.70) but only 4 points on presentation (58.16 vs 62.50), and score 38.54 versus 49.48 on overall rating. Reviewers flag 86.46% of AI proposals as AI-generated versus 39.06% of human papers. Even DeepSeek-V4-Flash, which surpasses the papers on soundness, presentation, and contribution, trails in feasibility and expected success.

• Proposals outside the human record are weaker, not overlooked. Both matched and non-matched proposal groups trail papers by the same 11 points in overall rating, but non-matched ones fall further behind in execution — the feasibility gap widens from 8 to 15 points and the expected-success gap from 15 to 26 — suggesting weaker directions rather than missed opportunities.

Methodology in Plain English

• Build two snapshots of each field. For each of four areas (machine learning systems, code generation, world models, automated research), the authors follow citations out from 20–22 hand-picked landmark papers to assemble a corpus of everything published through 2025 (88,184 / 94,674 / 50,271 / 30,264 papers). The 2026 reference is collected separately, screening landmark citations and topic-search results against a written area definition and removing surveys, covering January to August 2026 (5,188 / 3,375 / 1,867 / 1,136 papers).

• Run LLM agents as the ideation stage. An agent that can plan, search the corpus, pin papers, revise its draft, compress its history, and submit writes one proposal within 20 steps, citing only retrieved papers so its citations always lie in the frozen corpus. Four methods vary only what the agent knows: nothing beyond the corpus (free), one assigned topic (topic), a map of the area's task tags, method tags, and task–method pairs with their shares (landscape), or a view of peers' proposals in earlier rounds (frontier).

• Score citations with set overlap. Match-Cite takes the Jaccard similarity of citation sets, matches each document to its closest counterpart in both directions, and combines the two directional averages so a high score requires both reading like some human paper and covering the sources human papers use. Diversity-Cite multiplies a capped sufficiency term by a spread term, so citing more than a typical human paper earns nothing and reusing the same sources lowers the score.

• Score ideas with a slot-wise judge. An LLM judge (GPT-5.6 Luna, outside the evaluated families and validated against human annotators) compares idea summaries extracted under a fixed five-slot schema, comparing each document only against its five nearest neighbors retrieved over titles and abstracts. Slots are weighted by rarity (rarer slots like research plans count more), and Match-Idea is the weighted fraction of human slots covered by proposals while Diversity-Idea is one minus the weighted fraction a proposal shares with its peers.

• Check whether the gap is just sample size. Populate more agents and see whether matching converges toward the human level.

• Ask humans, blindly. Reviewers rate proposals and papers without knowing the source, including feasibility and expected success, and guess whether each document is AI-generated.

Why This Matters

Impact on research. The paper reframes what "AI scientist" progress should be measured against. Rather than rating ideas one at a time with unreliable judges, it anchors evaluation to what human communities actually published after a cutoff — a reference that grows over time and cannot be gamed by a single reviewer's taste. The diagnosis matters more than the headline number: agents reach the right problems and the right method vocabulary, but do not convert a recognized problem into the concrete mechanism a community pursues. That points future work at approach-level proposal quality rather than at broader literature retrieval or bigger sampling.

Real-world applications

• Research portfolio and funding decisions: a rolling comparison of machine-generated research directions against a field's subsequent publications gives program officers a population-level signal about where AI-proposed agendas diverge from community practice.
• Due diligence on AI-generated research artifacts: the striking finding that reviewers flag 86.46% of AI proposals but also mislabel a real 2026 paper as AI-generated (and miss AI proposals) argues for tooling that supports rather than replaces human screening.
• Evaluating and iterating on AI scientist systems: the free / topic / landscape / frontier ablation isolates how much of a system's apparent quality comes from being told a direction versus genuinely selecting one.
• Forecasting and trend analysis: the task/method/pair tag-coverage procedure offers a reusable way to quantify how a generated corpus distributes over an area's vocabulary relative to human output.

Industry relevance. Laboratories deploying AI research agents at scale need an outcome-anchored benchmark that does not saturate when models get newer or populations get bigger — the paper shows both levers fail here. The continuously updated leaderboard format also matters commercially, since it re-scores today's proposals against next year's publications and therefore cannot be fixed once and forgotten.

Future Directions

• Improve the conversion from problem to approach. Since agents already cover a community's tasks and methods but combine them differently and reach only about half of the human task–method pair mass even in the best method, the open question is what mechanism would let an agent commit to the specific approach a field goes on to pursue.

• Separate taste from leakage more cleanly. Four of six models may have seen the 2026 papers in training, and the paper handles this by including two pre-2026 models and by discussing the information asymmetry across the cutoff; a benchmark design that rules out contamination by construction is left open.

• Revisit the citation standard. The authors note that citation lists are an imperfect standard for reading behavior, and that idea judgments depend on a fixed five-slot schema and an LLM judge — both are targets for refinement.

• Test whether other agent designs or incentives help. The four methods vary only what the agent knows, not how it reasons; whether different planning, critique, or falsification loops (as in the case-study papers that co-evolve an adversarial falsifier) would raise approach-level matching is untested here.

• Extend beyond four areas and one cutoff. The cutoff "can be moved forward as new papers appear," and the leaderboard is designed to be continuously updated, so broadening to more fields and repeated yearly cycles is a natural next step.

Target Audience

Researchers building or evaluating AI scientist and research-agent systems; benchmark and evaluation designers interested in outcome-anchored metrics; science-of-science and bibliometrics researchers studying how research directions emerge; and research administrators, program officers, or industry lab leads who need to reason about where AI-generated research agendas do and do not align with a human community's actual trajectory. Readers wanting to reproduce or extend the benchmark should be comfortable with LLM agent pipelines and citation-graph construction.

Authors’ abstract

Research taste, the judgment of what to read, which problems to pursue, and which methods to try, increasingly shapes the work of AI scientists, yet it remains far less understood than their ability to execute research. We introduce IdeaTaste, a benchmark that scores an LLM by how closely the research proposals it writes from a literature snapshot frozen at 2025 match the work the same communities published in 2026. Our experiments span six contemporary language models and four research areas from established to emerging fields, covering more than 90{,}000 proposals in total. LLM agents fall well short of human communities in both the literature they cite and the ideas they state, reaching about 26\% of the human level on citation matching and only 18\% on idea matching. Their populations can cite as diversely as a community and take up its tasks and methods, yet rarely arrive at the concrete approaches and research plans it pursues. Neither richer field context nor larger populations substantially close this gap. In blinded expert review, AI proposals trail the 2026 papers least in presentation and most in expected success, echoing their gap in concrete approaches, and those whose ideas no human paper shares fare worst, suggesting weaker directions rather than overlooked opportunities. These results indicate that the main bottleneck in AI research taste lies less in finding the directions a community works on than in turning them into the concrete approaches it goes on to pursue. We release IdeaTaste as an open, continuously updated leaderboard (https://cxcscmu.github.io/IdeaTaste) that benchmarks contemporary AI proposals against future publication cycles.