The Pulse
AI Scientists Struggle to Reproduce Quantum Research
A study of AI research agents found failures across computational quantum science, including cases where papers, code and results conflicted. Its Residual Agents system improved reproduction rates by learning from expert-verified workflow r
AI.info Team · Written with AI · fact-checked ·
AI agents cannot reliably reproduce published quantum-computing results, but the papers are not always a dependable guide: some conflict with their own code or reported results. A study submitted to arXiv on November 2 by Yuxuan Du and five co-authors examines both sources of failure, testing agents against computational quantum research and using expert-verified repairs to improve one system.
The authors’ benchmark, OmniQ, turns published studies into tasks in which an agent must reconstruct a method and reproduce a reported result. Their follow-up system, Residual Agents, stores verified workflow repairs as reusable experience. The researchers say the approach lifts one model’s reproduction rate from 68% to about 85%.
OmniQ finds problems in papers as well as agents
OmniQ assesses whether an agent can recover a study’s method, implement it and reproduce a target result under a comparable computational budget. The authors first assembled 60 studies covering classical simulation, quantum algorithms and quantum-property characterization. Expert review identified inconsistencies in 18: some could be resolved from code or expert interpretation, while others left the reported result unreproducible from the available record.
The researchers corrected issues they could resolve, then excluded one unresolved case from the agent evaluation, leaving 59 studies. They tested five AI research systems and six third-party agent harnesses. That distinction matters to the paper’s central finding: a failed reproduction can reflect an agent’s limits, a flaw in a paper’s instructions, or a mismatch among the paper, code and results.
A code setting changes one published result
One example concerns a paper on quantum error mitigation. Its written method says a model selects its best split using one feature, but the released code uses all available features. Following the paper’s instructions loses the reported advantage; following the code recovers it.
A separate case showed a harder limit. For one quantum-kernel study, the paper describes a figure as measuring the variance of a kernel, while its code calculates the variance of a quantity used to form that kernel. The authors say expert inspection found the statistical error but could not establish which configuration generated the archived data, leaving the result unresolved.
Residual Agents learns from verified repairs
Residual Agents, or RA, breaks tasks into sub-plans, has executors carry them out, and uses a critic to check their results against the paper. When a step fails, the system can revise it or replan. Human experts review proposed updates before RA adds them to its stored experience, while the underlying model and orchestration remain fixed.
For development, RA used five challenging tasks; the other 54 were held out for evaluation. On those tasks, the team compared DeepSeek V4.1 Flash with and without RA using the same DeepSeek Harness and a 24-hour budget per task. The baseline succeeded on 68%; RA reached about 85% at an inference cost of $5.6 per task. The paper says that matches Claude Code with Opus 5, which reached about 85% at $16 per task.
Reproduction is not yet discovery
The authors write, Our study establishes paper-to-result reproduction as a foundation for developing autonomous AI scientists. Yuxuan Du, Hanqing Zhao, Xinbiao Wang, Yiming Lu, Kaining Zhang and Dacheng Tao, “Reimagining computational quantum research with AI scientists”
The paper presents RA as a system for improving reproduction, not as evidence that it has discovered new quantum algorithms. Its authors identify extending the method from verified reproduction repairs to verified algorithmic advances as a direction for future research.