The Pulse
AI Research Agents Match Just 18% of Human Ideas
The IdeaTaste benchmark found that language-model agents matched human research ideas at 18% of the human level, compared with 26% for citations. Its authors tested more than 90,000 proposals across four research areas and found the largest
AI.info Team · Written with AI · fact-checked ·
AI research agents matched human research ideas at only 18% of the human level in a new benchmark, while reaching about 26% on matching citations. The results, reported in the paper “IdeaTaste: Do LLMs Share the Research Taste of Human Researchers?”, show a wider gap in what agents propose than in the literature they draw on.
IdeaTaste looks ahead to 2026 papers
The authors compare proposals written from a research-literature snapshot frozen at 2025 with papers the same research communities published in 2026. The arXiv record lists the paper as submitted on Nov. 2, 2026. The benchmark covers machine learning systems, code generation, world models and automated research.
Across those areas, six language models produced more than 90,000 proposals. The researchers tested four ways of supplying field knowledge, ranging from the literature corpus alone to broader field context and proposals generated in earlier rounds. They compared agent populations with the level of matching among the human papers themselves.
The gap is in proposals, not just reading
IdeaTaste measures both whether proposals match human papers and how varied the proposals are, separately examining cited literature and stated ideas. The results show that agents can draw on literature and take up a community’s tasks and methods without often reaching the specific approaches and research plans in its papers. “The shortfall lies in what LLM agents propose more than in what they read,” write Shanshan Zhong, Chen Xu, Jingjie Ning, Zhengzhong Liu, Aviral Kumar and Chenyan Xiong.
No tested population reached half the human level on either citation or idea matching. More field context and larger populations did not substantially close the gap; the paper reports that populations grew as much as 17 times larger yet still leveled off well below the human level.
Reviewers saw the biggest gap in expected success
The authors also had reviewers assess proposals and published papers without knowing their source. Eleven reviewers with reviewing experience evaluated 192 documents, split evenly between AI proposals and 2026 papers; each document received two reviews. The assessment included expected success and feasibility as well as standard research-review dimensions.
In the blinded review, AI proposals trailed papers by 20 points on expected success and by four points on presentation. Reviewers identified 86% of the proposals as AI-generated, compared with 39% of the papers. Proposals with no matching human paper fared worst in reviewers’ assessment of expected success, which the authors interpret as evidence of weaker directions rather than overlooked opportunities.
A public benchmark for future research cycles
The authors’ conclusion is that agents more readily identify the directions a community pursues than develop the concrete approaches it adopts. IdeaTaste is released as an open, continuously updated leaderboard designed to compare current AI proposals with future publication cycles. The IdeaTaste leaderboard is the benchmark’s public home.