Can AI Agents Make Open-Ended Scientific Discovery? Evidence from Station

AI summary

Overview

Research area: Autonomous scientific discovery by multi-agent AI systems, evaluated in an open-ended (metric-free) research setting.

Technical level: Intermediate. The framing and results are accessible, but the tasks draw on reinforcement-learning interpretability, LLM output structure, RNN temporal representations, and VLM hallucination analysis.

Scope: The paper asks whether multi-agent AI systems can make progress on open-ended scientific questions where no objective progress metric exists, using an adapted multi-agent environment called Station across five tasks.

What This Paper Is About

Most AI-for-science systems have succeeded on problems with clear metrics or tightly prescribed pipelines, such as competition math or benchmark-driven algorithm discovery. This paper asks whether AI agents can instead work on open-ended research questions, where judging the scientific value of a partial result requires subjective human judgment. To test this, the authors extend Station—an open-world, multi-agent environment simulating a decentralized scientific ecosystem—with two mechanisms (a Supervisor agent and periodic Meta Reflection), then measure how much of the withheld findings from real published papers the agents can rediscover, and whether they can produce new findings on tasks with no oracle paper.

Key Contributions

1. An empirical study of autonomous discovery on five open-ended tasks—three rediscovery tasks built from ICLR papers and two exploratory tasks with no oracle paper—using comparative evaluation against baselines, ablations, and behavioral analysis.
2. An adaptation of Station from metric-driven tasks to open-ended research, adding a Supervisor agent that enforces minimum commitment periods on proposed research directions, periodic Meta Reflection every 50 ticks, and a Consolidation Agent that produces the final research report.
3. Evidence that the extended Station recovers withheld scientific findings at an average of 62.7% of partitioned sub-discoveries across three rediscovery tasks, versus 15.4% for Codex Multiagent-v2 and 14.4–20.6% for AI Scientist-v2, and that it makes findings on two exploratory tasks that coincide with concurrent human work.
4. Public release of source code and full agent dialogues to support further research on open-ended scientific discovery (code and experiment-data repositories are given in the paper).

Main Findings

• Average rediscovery: Station recovers 62.7% of the criteria on average across the three rediscovery tasks, compared with 15.4% for Codex Multiagent-v2 and 14.4–20.6% for AI Scientist-v2.
• Emergent planning in RL agents: Station recovers an average of 8.33 out of 11 sub-discoveries (75.8%), outperforming Codex Multiagent-v2 (6.1%) and AI Scientist-v2 (6.1–19.2%). Recovery is stronger for probing findings than for planning features or causal interventions; the agents fail to recover the oracle paper's finding that the RL agent plans bidirectionally.
• Low-rank structure in LLM outputs: Station recovers an average of 8.44 out of 12 sub-discoveries (70.4%), versus Codex Multiagent-v2 (33.3%) and AI Scientist-v2 (32.4–37.0%). Recovery is nearly complete for sequence-level structure and reusable relations but limited for functional consequences.
• Temporal representations in RNNs: Station recovers an average of 4.19 out of 10 sub-discoveries (41.9%), versus Codex Multiagent-v2 (6.7%) and AI Scientist-v2 (0.0–11.1%). Recovery is stronger for linear dynamics and spatial–temporal tradeoffs than for nonlinear dynamics; agents rarely recover explanations of how nonlinear models abruptly suppress outdated information.
• Ablation result: Removing both Supervisor guidance and Meta Reflection reduces sub-discovery recovery on the emergent-planning task from 75.8% to 57.9%, with a pronounced drop in intervention findings.
• Research persistence: The average duration of a research project drops from 67.0 ticks in Station to 55.5 ticks in the ablated Station.
• Meta Reflection behavior: Across 92 reflection events, 71.7% are followed by experiments that deepen or validate the existing research direction.
• Subliminal learning (exploratory): Agents find that trait transfer depends non-monotonically (inverted-U) on LoRA rank, matching a finding in concurrent work by Nief et al.; SVD shows retaining only a few leading singular modes of early-layer LoRA updates preserves most of the transferred trait, also matching that concurrent work; and training rank-128 LoRA adapters only in layers 0–13 restores cat preference, which the authors describe as a novel finding beyond the concurrent localization results.
• VLM hallucination (exploratory): Agents identify an intervention that multiplies the MLP outputs of layers 34 and 35 in InternVL3.5-8B-Instruct by 0.5 during prompt prefill, increasing correct answers from 28 to 35 on a 40-example visual-hallucination dataset. The principle and explanation closely align with concurrent work by Guo et al.
• Remaining gaps: Judgment misalignment and the streetlight effect persist. Asked to rate 24 agent research proposals that did not overlap with the oracle paper's sub-discoveries, one author of the emergent-planning oracle paper gave scores of 1 or 2 out of 5 to 75% of them, commenting that they focused on minor details.

Methodology in Plain English

The researchers took Station, an environment in which several AI agents (each backed by a different model) move between rooms, run experiments through a coding assistant, publish papers into a shared archive, exchange mail, and hold public discussions, and adapted it for questions with no scoring function.

Two additions target premature pivoting. A Supervisor agent—automatically assigned to a GPT-model agent once it has published a paper—cannot run experiments or publish, and instead reviews each agent's research proposal and holds it to a stated commitment period; agents can request permission to change direction, but approval is granted only in exceptional cases. Meta Reflection forces every agent, every 50 ticks, to pause and answer a self-evaluation prompt sampled from a predefined pool; these responses are always generated by GPT-5.5 regardless of the agent's underlying model, and the prompts ask agents to judge their work from a human researcher's perspective. A final Consolidation Agent turns the run's findings into an approximately 8,000-word research report, replacing the original Station's metric-based selection of the best algorithm.

For three rediscovery tasks, the authors built tasks from papers selected for oral presentation at ICLR 2025 and ICLR 2026. Agents received the main research question and the experimental setup but not the paper's findings, and internet access was disabled. Before any run, the authors partitioned each oracle paper's main findings into roughly ten sub-discoveries, and credited a sub-discovery only when the report stated a substantively similar finding with supporting evidence. Each of the three rediscovery tasks was run three times with different random seeds, each run lasting 300 ticks with six agents (three Gemini 3.1 Pro, one GPT-5.5, two Claude Opus 4.8). Because consolidation is stochastic, the Consolidation Agent was invoked three times per run, and each report was scored three times by a GPT-5.5 evaluator, giving nine scores per run; a blinded expert assessment showed high agreement with the automated evaluations. Four baseline configurations were compared (AI Scientist-v2 with each of three models, and Codex Multiagent-v2 with GPT), all given the same questions and setups and matched to Station's cumulative experiment time.

For two exploratory tasks—subliminal learning in LLMs and hallucination in VLMs—no oracle paper was designated. Findings were screened manually, compared with related work, and reported if novel or coinciding with concurrent work. One run was conducted per exploratory task.

Why This Matters

Impact on research. The paper shifts attention from benchmark-driven AI-for-science to the design of the research environment itself, arguing that a decentralized environment plus mechanisms that discourage early pivoting can let agents recover findings and produce results that overlap with concurrent human work. It also documents where agents still fail: they stop at probing-level descriptions rather than causal or mechanistic explanations, and they gravitate toward directions with a clear performance metric.

Real-world applications (the paper reports no deployed applications; these follow from the task domains and capabilities it demonstrates):

• Interpretability tooling: automatically probing and causally intervening on trained models to explain their internal computations.
• Model and training diagnostics: characterizing properties of LLM outputs and representations, and finding lightweight interventions such as the late-layer MLP attenuation that reduced VLM hallucinations.
• Fine-tuning practice: the subliminal-learning findings about LoRA rank and layer placement inform how adapters are configured when behavioral traits must or must not transfer.
• Accelerating early-stage exploratory research: generating candidate hypotheses and pilot evidence in fields where no obvious score exists.

Industry relevance. The result that a decentralized multi-agent environment substantially outperforms centralized baseline systems on open-ended tasks is directly relevant to organizations building autonomous research agents, and the paper's finding that agents can still waste effort on uninteresting directions, and that human judgment remains a gap, matters for anyone deciding how much autonomy to grant.

Future Directions

• Closing the judgment gap. Judgment misalignment and the streetlight effect remain unresolved; the authors suggest human guidance could narrow the gap through lightweight steering, but note that reliance on such guidance limits scalability, making improved environment design an important direction.
• Attributing contributions to individual components. The authors state that questions about the contributions of individual components are left open (Appendix F), so isolating the effects of the Supervisor, Meta Reflection, and the Consolidation Agent separately is unresolved.
• Tasks requiring greater agent autonomy. The temporal-representations task, which requires agents to train their own models rather than use provided checkpoints, produced the lowest recovery (41.9%) and is described as highlighting the challenges of tasks that give agents more freedom in choosing research directions.
• Deeper, mechanistic findings. Agents repeatedly fail to build on earlier discoveries to reach deeper explanations—for example, they miss the oracle paper's account of bidirectional planning and of abrupt forgetting in nonlinear RNNs—pointing to a need for mechanisms that support cumulative investigation.

Target Audience

Researchers and engineers working on autonomous research agents, multi-agent LLM systems, and AI-for-science pipelines; practitioners who need to evaluate whether an agent system can operate where no clear metric exists; and readers interested in the limits of current agents relative to human researchers, including those studying novelty assessment and research judgment.

Authors’ abstract

Recent AI systems have made rapid progress in scientific discovery when given well-defined metrics, but whether they can autonomously undertake open-ended scientific discovery remains unclear. We investigate AI's ability to tackle open-ended tasks in Station, an open-world environment in which multiple agents simulate a scientific ecosystem. To tackle challenges specific to open-ended tasks, we propose augmenting Station with two mechanisms: a Supervisor mechanism and periodic Meta Reflection, which encourage persistent exploration even when intermediate metrics are lacking. We construct open-ended tasks from three recent oral papers presented at ICLR. We give agents the main research question studied in each paper while withholding the paper's results and disabling web access. We then measure how many of the original findings-partitioned into individual criteria-agents rediscover. We find that Station rediscovers 62.7% of the criteria on average, compared with 15.4% for Codex Multiagent-v2 and 14.4-20.6% for AI Scientist-v2. Ablation and behavioral analyses indicate that adding the two mechanisms together improves research coverage and continuity. We further evaluate Station on two open-ended tasks without oracle papers and find that some of the discoveries made by the agents closely match discoveries reported by researchers after the knowledge cutoff date. Together, these results indicate that a suitable environment can enable agents to autonomously make meaningful progress in open-ended scientific discovery.