The Pulse
Study Finds AI Judges Favor AI-Assisted Research Proposals
A study of 70 student research proposals found no significant human-rated quality advantage for teams working with AI. GPT-4.1 scored AI-assisted proposals more favorably than human reviewers did, while proposal revisions improved ratings a
AI.info Team · Written with AI · fact-checked ·
GPT-4.1 rated research proposals developed with AI more highly than proposals produced by student teams without AI, even though human reviewers gave the two groups nearly identical scores. That split is the sharpest finding in a new study of how AI affects scientific ideas—not only by helping produce them, but by influencing how they are judged.
The paper, “Beyond Better Ideas: How Human-AI Collaboration Shapes Scientific Ideas and Human Perceptions,” was submitted to arXiv on October 22 by Qian Chen, Saatvik Pradhan and eight co-authors. The authors write: “AI collaboration shows no significant advantage in human-evaluated idea quality, while iterative revision consistently improves quality.” Their study covers 70 one-page proposals from 35 student teams, alongside evaluations by human reviewers and four large language models.
70 proposals, two rounds of review
The teams worked on 20 research questions in natural language processing and machine learning. Thirty-six proposals came from Human+AI teams and 34 from Human+Human teams; each team submitted an initial proposal and a revision one week later. Reviewers scored proposals on novelty, feasibility, clarity, broad applicability, correctness and overall quality, then ranked proposals addressing the same question.
Human reviewers found little difference in overall quality between collaboration groups: Human+AI proposals scored just 0.04 points higher on average, on a five-point scale, and the difference was not statistically significant. Revision, by contrast, was associated with a measurable improvement across the teams: average overall scores rose by 0.18 points from the first submission to the second. The authors caution that proposals from the two collaboration conditions addressed the same question in only one of the 20 cases, so their comparison does not isolate AI’s causal effect.
GPT-4.1’s preference diverged from human scores
The evaluator comparison tested GPT-4.1, Gemini 2.5 Flash, Claude Sonnet 4.6 and Qwen 2.5 72B against a human-reviewer panel. GPT-4.1 came closest to the panel’s numerical scores, with a mean absolute error of 0.48, but its ordering of proposals agreed less closely with the panel’s than individual human reviewers’ rankings did. The finding separates two questions often bundled together: whether an AI judge assigns similar scores and whether it picks the same proposals as stronger.
When the researchers compared collaboration conditions, GPT-4.1’s average across six ratings was 3.90 for AI-assisted proposals and 3.76 for proposals from Human+Human teams. The human panel’s corresponding average was 3.605 for each group. GPT-4.1’s overall-quality ratings also favored the AI-assisted group, while human overall scores differed only slightly. The largest gap by review criterion concerned broad applicability: GPT-4.1 rated AI-assisted proposals higher, while human reviewers rated them slightly lower.
Collaboration was hard to identify reliably
The paper also tested whether a model could infer how a proposal had been produced from its research question and methodology. A classifier fine-tuned on synthetic examples reached 65.7% accuracy on the 70 real proposals in one text format; the best results across tested formats and models were around 66% to 67%. The authors describe the signal as detectable but too weak for dependable attribution of an individual proposal’s collaboration history.
That result matters because a system’s ability to identify AI involvement could affect judgments about credit and disclosure. Here, however, the evidence does not support treating a model’s classification as proof of who—or what—contributed to a particular research idea. The proposals came from a student study, not a sample of grant applications or professional peer reviews.
Hands-on use shifted expectations in both directions
Questionnaires suggested that direct experience made participants more likely to see GenAI as helpful for research design, while the perceived speed-up fell short of what they had expected. The study reports increases in perceived helpfulness and trust, alongside a decrease in expected-versus-experienced speed-up. These comparisons are descriptive: the available pre- and post-study responses were not fully paired, and the authors caution against reading them as proof of individual-level causal change.
The paper therefore points to a tension in AI-assisted science: AI can contribute to creating a proposal and then act as a judge of its quality. In this study, the AI judge’s higher ratings for AI-assisted work did not correspond to a human-rated quality advantage. The result is a warning about evaluation design, not evidence that AI-assisted research is inherently weaker—or that human judgment is free of its own limits.