The Pulse
Ai2 Benchmark Finds LLM Novelty Scores Flip With Prompts
Ai2 tested six language models as judges of scientific research ideas and found that small prompt changes could sharply alter their verdicts. On one test, GPT-5.4’s accuracy fell from 93.5% to 40.9% when the prompt changed.
AI.info Team · Written with AI · fact-checked ·
GPT-5.4’s accuracy at picking the more novel idea in a pair plunged from 93.5% to 40.9% after a prompt change, according to a benchmark Ai2 released on October 27. The model saw the same pairs in both tests; the wording changed whether it was told that reviewers had identified one idea as novel. At 40.9%, its score fell below the 50% expected from a random choice between two options.
The result points to a problem for automated research systems that use language models to judge whether a proposed idea adds something new. Ai2 tested six models and found that their judgments could shift with prompt wording, reasoning budgets and access to related research. The findings appear in a study titled Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation, alongside a benchmark, data and code.
ICLR reviews supplied the novelty labels
Ai2 built its benchmark from peer reviews of ICLR 2026 papers, using reviewers’ explicit comments about originality rather than treating acceptance as proof that a paper was novel. Its higher-novelty set contains 154 ideas from accepted papers in the top-rated 10% of their research areas, with reviewers praising novelty and none disputing it. The lower-novelty set contains 145 ideas from rejected papers in the bottom 10%, whose reviewers raised originality concerns without any reviewer praising novelty.
For a second test, the researchers paired the 154 higher-novelty human ideas with 154 ideas produced by a simple language-model generator. They treated the generated ideas as less novel, but identify those comparisons as weak labels: the researchers did not verify the novelty of every generated idea. The generator received a research area and an instruction to propose an idea, with no literature search, tools, iteration or feedback.
Ai2 also asked an expert to assess 30 human–generated pairs, selecting cases heavily weighted toward errors by the strongest model judge. The expert preferred the human idea in all 30 and judged none of the generated ideas novel. That check supports the researchers’ comparison, but its small, deliberately selected sample is not a broad independent evaluation of every benchmark label.
A few words changed more than half of GPT-5.4’s choices
Across the six judges—GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6—the researchers varied prompts, reasoning budgets, access to related papers, formatting and the number of judgments used to reach a verdict. The largest performance shifts came from prompt changes. In the GPT-5.4 test, the model changed its answer on more than half of the same human–generated pairs when the prompt no longer said reviewers had found exactly one idea novel.
The shift was smaller on pairs of human-authored ideas with stronger and weaker evidence of novelty: the same prompt change moved accuracy by about seven percentage points overall. Claude Opus 4.6 still shifted by 21 points on that set. Ai2 also found no prompt that helped every judge; removing a warning against treating incremental contributions as novel improved some models’ results and hurt others.
The authors write: “Retrieval and larger reasoning budgets help little, and two purpose-built novelty evaluators are outperformed by our cheapest prompted baseline.” The result captures a central caveat in the study: adding tools or more reasoning does not guarantee steadier judgments.
More expensive evaluators scored lower
On the human–generated set, a Claude Opus 4.6 prompt using low reasoning and a majority vote across three calls scored 0.854 macro-F1, at a total evaluation cost of $2.59. Ai2 compared it with Idea Novelty Checker, a literature-retrieval and reranking system that scored 0.642 at $30.66, and Sakana AI’s AI Scientist novelty evaluator, which scored 0.772 at $82.05. Both purpose-built systems also used Claude Opus 4.6, with one call per decision.
Macro-F1 measures performance across the novel and non-novel categories. The cost figures reflect the evaluations reported in the study and API prices at the time, not a universal price for using the systems. Ai2’s broader finding is narrower than a claim that models cannot assess novelty: it is that results can depend heavily on how the task is framed, and that the tested retrieval and specialized systems did not reliably solve that problem.
The benchmark’s labels have limits
Reviewer judgments provide evidence about originality, but they are not a direct measurement of whether an idea is truly unprecedented across the literature. Ai2’s second dataset makes that uncertainty more pronounced because it assumes basic generated ideas are less novel than human ideas drawn from strongly reviewed ICLR papers. The researchers say disagreement between reasonable prompts can itself reveal instability, even where the correct answer is uncertain.
That distinction matters when novelty scores are used to compare idea-generating systems or filter proposals before researchers commit time to them. A score that changes with a few words in the prompt may reflect the judge’s setup as much as the idea being assessed. In Ai2’s GPT-5.4 comparison, the same pairs received a 93.5% accuracy result under one prompt and 40.9% under another.