The Pulse
Google Puts ScientistTwo on Display at COLM 2026
Google’s COLM 2026 program listed a live demo of ScientistTwo, a multi-agent system designed to run machine-learning research from a problem statement through experiments and paper drafting. Its authors report improvements on 86 of 107 rese
AI.info Team · Written with AI ·
ScientistTwo improved 86 of 107 research problems in Google researchers’ evaluation, an 80.4% success rate the company put before attendees at COLM 2026. Google’s conference program listed a live, half-hour booth demonstration for October 6 in San Francisco, describing a system that takes a research paper, identifies its limitations and produces a new paper with executable code.
From a paper’s limits to a tested proposal
The system’s authors describe a workflow that begins with a scientific problem and a human-provided challenge. ScientistTwo establishes a baseline, generates hypotheses, tests promising ideas, runs ablation studies to examine which components contribute to gains, and assembles its findings into a manuscript and codebase. The paper describes those stages as a collaboration among specialized AI agents, with no human intervention during the research cycle.
“In this paper, we introduce ScientistTwo, a fully autonomous multi-agent framework designed to realize this vision,” write Jaehyun Nam et al., authors of “ScientistTwo: Pioneering the Human Knowledge Frontier with Autonomous AI”. The authors say the system also uses simulated review and rebuttal: reviewer agents critique drafts, while a rebuttal agent can run additional experiments in response.
What the 86 successful runs mean
The evaluation draws on 107 research problems from papers associated with ICLR, ICML and NeurIPS. The authors report that ScientistTwo advanced 86 of them and achieved a 25.2% average relative performance gain over the original human-designed state-of-the-art methods. The 80.4% figure refers to the share of target problems the system advanced, not to papers accepted by those conferences.
To assess manuscript quality, the team used two automated review systems: ScholarPeer and the Stanford Agentic Reviewer. ScientistTwo’s generated papers received acceptance rates of 91.9% and 72.1%, respectively, under those systems’ evaluations. Those numbers are automated judgments, not records of acceptance at ICLR, ICML or NeurIPS; the paper compares them with scores for human-authored papers accepted at those venues.
Research cycles cost about $3,765
The reported gains come with a significant compute bill. For a subset of 33 problems drawn from NeurIPS 2025 papers, the authors estimate an average of two to three days for a complete research cycle and an average cost of $3,765, including token and virtual-machine expenses. They say the largest share of time and cost goes to iterative idea refinement and experiments.
That price and turnaround put the system’s practical scope in sharper view: ScientistTwo automates a long sequence of work, but each run still depends on computing resources and benchmarks that give its agents measurable targets. The paper also describes its main evaluation as research challenges drawn from machine-learning conference papers, rather than evidence that the system can independently select and solve any scientific problem.
The authors draw a line at major breakthroughs
The researchers say the results do not yet show ScientistTwo producing the kind of conceptual work that earns conference spotlight or oral presentation status. They describe its current strength as improving existing methods, and point to expanding beyond local algorithmic refinements as a next research challenge.
At COLM, Google’s schedule placed the ScientistTwo session at booth 107 on October 6, alongside the company’s broader slate of research talks and demonstrations. The system’s benchmark results are the authors’ own evaluation; whether its proposed methods hold up through independent human review and reproduction remains a separate test.