FML-bench: Benchmarking Machine Learning Agents for Scientific Research

AI summary

Overview

Research area: Natural language processing and machine learning, at the intersection of LLM-based autonomous agents and scientific research automation (benchmarking and evaluation).

Technical level: Intermediate. The paper is a benchmark and evaluation study; understanding it requires familiarity with ML research concepts (domain generalization, contrastive learning, causal inference, membership inference) and with how LLM coding agents operate, but not deep mathematical background.

Scope: The paper introduces FML-bench, an 8-task benchmark built on real research repositories, plus new process-oriented metrics — chiefly Exploration Diversity — and uses them to compare three automatic research agents.

What This Paper Is About

Most existing benchmarks for machine learning agents treat the problem as engineering: they use Kaggle-style competitions, ask agents to build data pipelines and submissions, and score them only on final results and compute cost. This misses the scientific research setting, where the goal is to formulate and test a methodological idea that improves on a known baseline. The paper's goal is to build a benchmark of fundamental ML research problems, together with metrics that describe how an agent researches (not just what score it reaches), and to use both to discover what makes a research agent effective.

Key Contributions

1. FML-bench, a benchmark of 8 diverse, fundamental ML research tasks — generalization, data efficiency, representation learning, continual learning, causality, robustness, privacy, and fairness — each built on a widely recognized academic repository with a provided baseline, rather than on application-oriented competition tasks.

2. Process-oriented evaluation metrics, notably Exploration Diversity, which measures the dispersion of code embeddings (extracted with GraphCodeBERT) around their centroid across a research run, plus Step Success Rate and Step Completion Rate for agent reliability, alongside standard Performance and Cost.

3. A unified input-output interface that handles repository diversity: the agent receives a task description, the full repository, suggested files to modify, protected read-only evaluation code, a command list for experiments, baseline performance, and target metrics; a post-processing module converts heterogeneous repository outputs into a standardized format.

4. Empirical findings from evaluating TheAIScientist, AIDE, and Claude Code, showing that broad exploration strategies and higher exploration diversity are associated with stronger research outcomes.

Main Findings

• TheAIScientist with Gemini-2.5-Pro ranked first in 4 of 8 tasks, the strongest overall configuration; AIDE with Gemini-2.5-Pro ranked second, securing top results in 2 of the 8 tasks.

• Every agent improved on the baseline somewhere, but not everywhere. AIDE with either GPT-5 or Gemini-2.5-Pro failed to improve on the baseline for Generalization (0.2254, matching the baseline of 0.2254) and Data Efficiency (0.6547, matching the baseline of 0.6547). Claude Code matched the baseline on Generalization (0.5036) and on Fairness and Bias (0.3787), and only marginally improved Data Efficiency (0.6571 vs. 0.6547 baseline).

• Large gains were possible on several tasks. The largest single jump reported is Continual Learning, where TheAIScientist with Gemini-2.5-Pro reached 0.7808 against a Synaptic Intelligence baseline of 0.2710. Privacy also moved far, with TheAIScientist and Gemini-2.5-Pro reaching 0.1750 (lower is better) against a baseline of 0.8114.

• Agents with broad exploration show higher exploration diversity. Mean exploration diversity was 24.66 for TheAIScientist, 23.50 for AIDE, and 8.75 for Claude Code. With the LLM held fixed, higher diversity tracked better outcomes.

• Diversity correlated positively with performance on five of eight tasks. Correlations reported were 0.8374 (Continual Learning, p = 0.0002), 0.6287 (Data Efficiency, p = 0.0121), 0.5292 (Fairness, p = 0.0517), 0.4418 (Generalization, p = 0.1137), and 0.4036 (Representation Learning, p = 0.1358). Causality (-0.2036), Robustness (-0.3846), and Privacy (-0.1286) showed negative correlations.

• Exploration diversity was not decisive on its own. When the agent was held fixed, performance also depended on other LLM capabilities such as reasoning and code-writing ability.

• Reliability differed sharply. Step Completion Rate was 1.00 ± 0.00 for both TheAIScientist configurations, 0.54 ± 0.38 for AIDE with GPT-5, 0.69 ± 0.24 for AIDE with Gemini-2.5-Pro, and 0.07 ± 0.06 for Claude Code. Step Success Rate ranged from 0.64 ± 0.30 (AIDE with GPT-5) to 0.83 ± 0.27 (TheAIScientist with GPT-5) and 0.83 ± 0.32 (Claude Code).

• Cost patterns were counterintuitive. Claude Code, the weakest performer, used the most tokens (9.06M ± 3.64 total; 3.10M ± 4.34 per step). TheAIScientist used 6.04M ± 2.18 total (0.06M ± 0.02 per step) and AIDE 5.45M ± 2.51 (0.05M ± 0.03 per step). Token usage for AIDE is marked N/A because it does not support recording it. Per-step time was similar across agents (roughly 10–14 minutes), but total time was much shorter for AIDE (5.46 ± 6.92 hours for GPT-5, 9.49 ± 7.24 for Gemini-2.5-Pro) and Claude Code (1.18 ± 1.80 hours) than for TheAIScientist (9.42 ± 5.25 and 11.74 ± 8.37 hours) — attributed to premature termination.

• Failure modes were diagnosed. AIDE sometimes generated new classes or components never integrated into execution ("shallow edits"), producing no improvement — likely because it only supports iterative modification of a single file. AIDE also terminated early when its cloud-based commercial variant Weco failed; Claude Code terminated early by its own internal reasoning.

• Behavioral case analysis. In worst cases, TheAIScientist strongly favored algorithmic modifications (56.2%) with zero implementation errors, AIDE showed higher implementation issues (31.2%), and Claude Code showed elevated parameter configuration rates (37.5%). By LLM, Gemini-2.5-Pro showed strong algorithmic modification tendency (68.8%) and GPT-5 primarily structural modifications (50.0%). In best cases, GPT-5 exclusively prioritized algorithmic modifications (100.0%) and Gemini-2.5-Pro showed the most diversified approach with data augmentation efforts (28.6%).

Methodology in Plain English

The researchers started from the observation that trustworthiness and capability in ML systems depend on recurring challenges — robustness, generalization, fairness, privacy, data efficiency, representation learning, causality, and continual adaptation — and selected one task per challenge. For each task they chose a dataset and a classic baseline method under two rules: the baseline must train and evaluate within 2 hours, and the baseline must sit below the theoretical optimum so there is headroom to improve. Baselines were built on real research repositories (DomainBed, Easy-Few-Shot-Learning, Lightly, Continual-Learning, CausalML), and agents could modify code but not the protected evaluation files.

To make one benchmark work across such different repositories, they treat the full execution sequence — training and evaluation commands — as a single input unit, handing the agent a command list rather than assuming a fixed script name or pipeline. A post-processing module normalizes the varied outputs into a standard metric format.

For evaluation, they run each agent for three independent rounds with a fixed budget of 100 total steps per round, and select the best result across rounds on the test set. Beyond the best final score, they compute exploration diversity by embedding each iteration's codebase with GraphCodeBERT and measuring the mean distance of those embeddings from their centroid. They note that GraphCodeBERT captures data-flow and code structure, not surface tokens, so logically identical code with renamed variables counts as low diversity. They then correlate diversity with performance, negating scores for lower-is-better tasks so positive correlations mean better results with more exploration.

The three agents differ in strategy: TheAIScientist explores broadly and in parallel, AIDE uses hierarchical tree-based search balancing exploration and exploitation, and Claude Code was prompted into linear sequential refinement. TheAIScientist and AIDE were run with GPT-5 (2025-08-07) and Gemini-2.5-Pro (2025-06-17); Claude Code used its native Opus-4.1 (2025-08-05).

Why This Matters

This paper shifts agent evaluation from "did it beat the leaderboard" toward "how does it actually do research," which is a more faithful proxy for scientific usefulness and exposes failure modes (shallow edits, premature termination) that final scores alone hide. It also shows that a general-purpose coding agent is not automatically a good research agent: Claude Code used the most tokens while performing worst, completing only 0.07 of its assigned steps.

Real-world applications:

• ML research acceleration. Teams could use broadly exploring agents to generate and cheaply test candidate methodological changes (e.g., new regularization for continual learning) before committing researcher time.

• Autonomous experimentation platforms. Labs building self-driving labs or automated ML pipelines can use the reliability metrics (Step Success Rate, Step Completion Rate) as deployment gates before trusting an agent with unattended runs.

• Model auditing for trust properties. Benchmarks covering privacy, fairness, robustness, and causality give a structured way to check whether an automated method degrades or improves a model's safety-relevant behavior on datasets such as COMPAS and poisoned MNIST.

• Agent product evaluation. Vendors of research or coding agents can benchmark on fundamental research tasks rather than only application-style competitions, giving buyers a more relevant signal.

Industry relevance: The cost findings matter commercially — token and wall-clock totals varied widely across agents for the same tasks, and the paper argues dedicated research agents are more suitable than general-purpose agents in both performance and token efficiency. The unified input-output interface also lowers the barrier to adding new repositories to a benchmark, which is useful for organizations that want internal, domain-specific evaluations.

Future Directions

• Understand the negative correlations. Diversity correlated negatively with performance on Causality, Robustness, and Privacy, and positively on the other five. Explaining why broad exploration hurts in these specific settings is an open question the paper raises but does not resolve.

• Fix single-file editing limits. AIDE's inability to improve Generalization and Data Efficiency is attributed to its restriction to modifying one file per iteration; extending agents to multi-file repositories is a clear next step.

• Address premature termination. Both AIDE and Claude Code stopped early for different reasons (infrastructure failure and internal model reasoning), so reliable stopping criteria and recovery mechanisms are needed.

• Separate agent design from LLM capability. The paper finds that with the LLM fixed, diversity tracks performance, but with the agent fixed, other LLM abilities dominate. Disentangling these factors requires more controlled cross-product experiments.

• Extend the benchmark. The authors state their unified interfaces enable extensibility across repositories, datasets, and baselines, inviting the community to add tasks beyond the current 8.

Target Audience

Researchers and engineers building or evaluating autonomous ML research agents, and benchmark designers interested in process-level rather than outcome-only evaluation. It is also relevant to ML practitioners who want to understand where LLM agents can genuinely contribute methodological improvements versus where they fail (generalization, data efficiency), and to teams making procurement decisions about coding and research agents based on cost and reliability data. Readers need comfort with ML research terminology; the paper is accessible to graduate students and applied scientists rather than to complete beginners.

Authors’ abstract

Large language models (LLMs) have sparked growing interest in machine learning research agents that can autonomously propose ideas and conduct experiments. However, existing benchmarks predominantly adopt an engineering-oriented perspective: they emphasize application-oriented tasks and evaluate primarily on final performance and computational cost, overlooking agents' research processes and limiting assessment of their capabilities in scientific research settings. To more comprehensively evaluate agents in scientific research settings, we introduce FML-bench, a benchmark comprising 8 diverse and fundamental ML research tasks, and further propose complementary metrics, notably Exploration Diversity, which quantifies the variance of proposals across iterations and reveals how exploration patterns influence research outcomes. We evaluate state-of-the-art research agents on FML-bench, showing that agents employing broad exploration strategies exhibit higher exploration diversity and achieve superior performance, and that exploration diversity positively correlates with performance improvements across multiple tasks. We hope these findings and our benchmark inform future agent design and support the community in further investigating agent behavior. Our benchmark is available at https://github.com/qrzou/FML-bench.