FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights

AI summary

Overview

• Research area: Evaluation of autonomous AI research agents — specifically benchmarks that test whether LLM-powered agents can carry out a full cycle of empirical machine learning research.
• Technical level: Advanced. The paper assumes familiarity with LLM agent frameworks, benchmark design, claim-level evaluation metrics, and research workflow concepts, though its central idea can be grasped without that background.
• Scope: The paper introduces FIRE-Bench, a benchmark that gives AI agents only a high-level research question from a published study and scores whether they can independently design, run, and interpret experiments to rediscover that study's documented empirical finding.

What This Paper Is About

Existing ways of evaluating AI research agents either ask them to write whole papers judged by another LLM, or reduce them to chasing a single leaderboard number. Neither reliably verifies whether an agent actually reasoned its way to a correct scientific conclusion. FIRE-Bench addresses this by turning recent, peer-reviewed empirical papers into verifiable "rediscovery" tasks: the agent receives the research question but not the original experimental design, implementation, or conclusion, and its final written claims are scored against the paper's documented findings.

Key Contributions

1. Constrained rediscovery as a new evaluation paradigm. FIRE-Bench occupies a previously empty point in benchmark design space, combining four properties at once: full-cycle execution (plan → code → execute → conclude), insight-driven evaluation of a scientific hypothesis, grounded reference-based scoring against a published result, and room for methodological exploration. The paper contrasts this with method-replication benchmarks (e.g., PaperBench, LMR-Bench), metric-driven discovery benchmarks (e.g., MLAgentBench, MLE-Bench, MLRC-Bench), and automated paper-generation systems (e.g., The AI Scientist, Agent Laboratory).

2. A research-problem tree abstraction with an automated extractor. Each source paper is formalized as a hierarchical tree with a root node (overarching research question), intermediate nodes (progressively narrower subproblems), and leaf nodes (fully specified experimental tasks defined by a dataset, method or model, and evaluation criteria, grounded in reported figures or tables). Extraction is automated with a fixed-prompt LLM extractor instantiated with gpt-5 Pro at greedy decoding (temperature 0), producing structured JSON. The formalism was validated on 100 papers across more than 10 ML subfields.

3. A diagnostic error framework for agent failures. The paper defines a taxonomy attributing each incorrect claim to one of four research-workflow stages — Research Planning, Implementation, Experimental Execution, and Conclusion Formation — with 16 total error categories.

4. A benchmark release and contamination analysis. FIRE-Bench releases 40 fully executed tasks plus 60 additional parsed papers for community evaluation, and includes an analysis that stratifies agent performance by task difficulty and model knowledge cutoff to probe for data contamination.

Main Findings

• All agents perform poorly in absolute terms. The strongest system, Claude Code (Claude-4-Sonnet), reached a claim-level F1 of 46.7 (±23.4), with precision 52.1 (±26.1) and recall 48.3 (±24.8). Codex (gpt-5-medium) reached F1 41.9 (±25.4), OpenHands with gpt-5 reached 37.9 (±23.0), and OpenHands with o4-mini reached 31.9 (±17.6). The abstract states that even the strongest agent stays below 50 F1.

• Run-to-run variance is very high. Each task-agent pair was executed three times. Standard deviations were high for nearly all pairs — for example, OpenHands (o4-mini) scored 57.0 ± 40.5 on Lost in the Middle, and Claude Code scored 66.7 ± 47.1 on Awareness Detection. The median coefficient of variation across tasks was 0.37 even for Claude Code, and exceeded 0.6 for the OpenHands variants, with standard deviations reaching or exceeding the mean on up to a third of tasks for weaker backbones.

• Task structure predicts success. Agents did best on procedurally direct tasks where the evaluation objective is explicit and the workflow is largely predetermined: Lost in the Middle (best observed F1 91.7), Persona with Catch (88.6), CoT Without Prompting (82.6), and Hallucination Snowballing (80.9). Performance degraded sharply on tasks requiring controlled or counterfactual experimental design.

• Control-based experimental design is a consistent failure point. In the LLM Racial Bias in Medicine task, the human ground truth required stripping racial indicators to build a race-free baseline and then reintroducing labels selectively. Every evaluated agent skipped the baseline step and injected race labels into notes that still carried latent demographic cues. OpenHands (gpt-5) and Claude Code each scored 0.0 ± 0.0 F1, and all four agents scored at or below 34.2.

• No single agent dominates. Claude Code achieved the best observed result on 13 of the 30 core tasks, Codex on 9, and OpenHands (gpt-5) on 6. Only 4 of 30 tasks saw all four agents reach F1 ≥ 50, and no task had all agents below 20. Several tasks showed gaps of 30 or more F1 points between the best and runner-up (e.g., CoT Without Prompting: Claude Code 82.6 versus OpenHands (gpt-5) 26.4), indicating agent-specific rather than purely backbone-driven strengths.

• Upgrading the backbone helps within a framework. Swapping OpenHands's backbone from o4-mini to gpt-5 improved average F1 by 6.1 points (31.9 to 37.9).

• Errors cluster in planning and conclusion formation. Across agents, error distributions were broadly similar, with failures in Research Planning (such as Method Deviation and Goal Deviation) and Conclusion Formation (unsupported or overgeneralized claims) accounting for the majority of errors, rather than Implementation or Experimental Execution.

• False positives are mostly clear errors, not alternative valid science. Contradictory and Unrelated claims accounted for 76.4% to 95.0% of false positives depending on the agent, while Alternative conclusions (plausible but non-aligned) comprised only 4.5% to 10.9%. Per-agent breakdowns: OpenHands (o4-mini) 42.0% Contradictory, 47.7% Unrelated, 5.7% Overgeneralized, 4.5% Alternative; OpenHands (gpt-5) 66.7/28.3/0.0/5.0; Codex 70.9/14.0/4.7/10.5; Claude Code 65.5/10.9/12.7/10.9.

• Cost and performance are positively associated, with one outlier. Codex (gpt-5-medium) reached F1 41.9 at a total cost of $2.21, roughly 5 times cheaper than Claude Code ($12.67 for F1 46.7). Within OpenHands, upgrading from o4-mini to gpt-5 raised total cost by 21% ($8.90 to $10.74) while adding 6.1 F1 points. Codex's efficiency reflected shorter action sequences and lower token usage rather than a weaker backbone. Tasks requiring longer reasoning chains (e.g., LLMs Lack Self-Correction, ICL from Repetition) were the most expensive across agents.

• Difficulty stratification behaves as expected. Tasks were labeled Easy (rubric score 3–4), Medium (5–6), or Hard (7–9) based on Conceptual Decomposition, Confound Control, and Analysis Complexity, each scored 1–3. Performance showed a clear monotonic relationship with difficulty, supporting the measure's validity.

• No strong contamination signal. Comparing pre- versus post-cutoff tasks within difficulty levels (knowledge cutoffs 2024-06-01 for o4-mini and 2024-09-30 for gpt-5) showed no consistent advantage for pre-cutoff tasks. On Hard tasks, both OpenHands variants scored higher after cutoff (15.4 to 24.8 for o4-mini; 22.6 to 31.0 for gpt-5). Medium tasks were mixed (o4-mini 31.9 to 33.6; gpt-5 44.5 to 23.1). Easy tasks declined for o4-mini (58.9 to 42.1) while gpt-5 stayed relatively stable. The authors describe this as suggesting the absence of a strong signal rather than proving none exists.

• Cross-domain results are similarly weak. On the 10-task extension (5 computer vision and vision-language modeling papers, 5 neural network analysis papers), average F1 fell below 25 for all three agents evaluated (Claude Code was omitted due to budget), with standard deviations frequently reaching 50–100% of the mean (e.g., Neural Collapse at 33.3 ± 47.1; MaxSup at 41.8 ± 29.7). A new pattern emerged where self-contained tasks that let agents generate their own test inputs scored higher; the paper's description of this pattern is truncated in the available content.

Methodology in Plain English

The researchers built the benchmark by working backwards from published results. They started with a keyword search over 2024 and 2025 proceedings from ICLR, ICML, and NeurIPS, then used an LLM-based classifier (gpt-4o-mini) to keep only papers whose main contribution is empirical analysis of LLM behavior, reducing the pool to about 50 candidates. Two authors then manually reviewed them and kept only papers meeting three criteria: all experiments use publicly available datasets and models, the core experiments fit within 24 hours on modest hardware (such as an 80GB A100 GPU) without large-scale training, and the findings are specific and verifiable through explicit figures or tables. This yielded 30 papers.

To turn each paper into a task, they parsed its logical structure into a research-problem tree. They picked a leaf node corresponding to a main figure or table, then used its parent node — a higher-level research question less prescriptive about experimental details — as the prompt given to the agent. The agent also received the experimental scope (such as datasets) and evaluation criteria, but not the original implementations or conclusions. The recorded result at the leaf node became ground truth. The authors argue this parent-node framing is strictly harder than giving the leaf node, which would reduce the task to replication.

Evaluation is claim-level. Both the agent's final conclusion and the ground-truth text (the figure or table caption plus relevant prose) are broken into atomic verifiable claims by an automated extractor using gpt-5.2. Each agent claim is then checked against the ground-truth claim set by an LLM-based semantic entailment classifier (the same gpt-5.2 judge for all agents). Matching claims count as true positives; unsupported agent claims are false positives; unmatched ground-truth claims are false negatives. From these, the paper computes precision, recall, and F1, following the claim-centric paradigm of RAGChecker. Human validation on 33% of reference instances produced precision 0.95, recall 0.86, and F1 0.89.

Agents ran in a sandbox through their command-line interfaces on a GPU node with eight 80GB A100 GPUs, each with an instruction file and provided datasets in the working directory. The team deliberately did not pre-install Python packages, treating environment setup as part of the agent's capability. Each task-agent pair ran three times; no hard runtime limit was imposed, though most runs finished within an hour. Experiments were conducted primarily in August 2025. For error analysis, an LLM annotated each erroneous claim with a pipeline stage, an error type from the fixed taxonomy, and a rationale citing trajectory evidence; all annotations were then human-verified by the authors.

Why This Matters

Impact on research. The paper argues that autonomous research agents cannot be trusted until their discoveries can be verified. Because genuinely novel findings need expensive real-world validation, existing benchmarks resort to LLM-as-judge scoring of generated papers or single leaderboard metrics — both coarse proxies. FIRE-Bench offers a middle path: open-ended discovery with an objective, reference-grounded scoring signal. Its diagnostic error framework further shifts evaluation from "did the agent succeed" to "at which stage of the research process did it fail," which is more actionable for building better agents.

Real-world applications:

• Automated ML experimentation platforms. Teams deploying agents to explore model behaviors or run ablation studies can use FIRE-Bench-style tasks as a readiness check before trusting agent-produced conclusions.
• Internal research reproduction and auditing. Organizations can adapt the constrained-rediscovery framing to verify whether an agent's claimed result matches a known published outcome, without handing over the original methodology.
• Agent development and iteration. Error-stage attribution (Planning, Implementation, Execution, Conclusion Formation) gives builders a targeted signal about which component to improve, rather than a single opaque score.
• Research tooling for scientists. The difficulty-stratified task set and the cost data (for example, Codex delivering F1 41.9 at $2.21 versus Claude Code's $12.67 for F1 46.7) give practitioners a concrete basis for choosing agent configurations under budget constraints.

Industry relevance. The finding that failures concentrate in planning and conclusion formation — not in coding or execution — is directly relevant to companies building autonomous R&D agents, since it suggests that raw coding capability is not the bottleneck. The cost-efficiency analysis also matters commercially: the observed Pareto-efficient outlier shows meaningful performance need not require the most expensive agent. The paper's release of 60 additional parsed papers as a living benchmark signals an intent to keep evaluation current as models and research literature evolve.

Future Directions

• Web retrieval remains an open problem. The authors note that agents could in principle search for the source paper instead of doing their own experiments. Trajectory inspection showed this did not occur in their runs, and they note blacklisting webpages is technically infeasible for proprietary agents; they explicitly leave a systematic treatment to future work.
• Improving planning and conclusion formation. Since these two stages dominate failures, the obvious next step is agent architectures or training that target experimental design and evidence-grounded claim generation, rather than coding skill.
• Stronger contamination controls. The cutoff-based analysis is described as coarse: knowledge cutoff dates are approximate, training data composition is unknown, and per-category sample sizes are limited. More rigorous methodology is needed to confirm or rule out memorization effects.
• Scaling and extending coverage. The core analysis rests on 30 tasks, with a 10-task cross-domain extension (for which Claude Code was omitted due to budget) and 60 papers released as a parsed pool. Expanding and continuously refreshing the benchmark, and evaluating more agent-backbone combinations across domains, are natural extensions.

Target Audience

This paper is most valuable to researchers and engineers building or evaluating autonomous research agents, and to benchmark designers interested in objective, reference-grounded evaluation of open-ended discovery. ML practitioners considering deploying coding agents for experimental work will find the cost and failure-mode data practically useful. Science-of-AI and AI-for-science researchers will benefit from the research-problem tree formalism and the diagnostic error taxonomy. Readers without background in LLM agents can still follow the core argument, but the full methodology assumes comfort with claim-level precision/recall/F1 evaluation, agent sandboxing, and experimental design concepts such as counterfactual controls.

Authors’ abstract

Autonomous agents powered by large language models (LLMs) promise to accelerate scientific discovery end-to-end, but rigorously evaluating their capacity for verifiable discovery remains a central challenge. Existing benchmarks face a trade-off: they either heavily rely on LLM-as-judge evaluations of automatically generated research outputs or optimize convenient yet isolated performance metrics that provide coarse proxies for scientific insight. To address this gap, we introduce FIRE-Bench (Full-cycle Insight Rediscovery Evaluation), a benchmark that evaluates agents through the rediscovery of established findings from recent, high-impact machine learning research. Agents are given only a high-level research question extracted from a published, verified study and must autonomously explore ideas, design experiments, implement code, execute their plans, and derive conclusions supported by empirical evidence. We evaluate a range of state-of-the-art agents with frontier LLMs backbones like gpt-5 on FIRE-Bench. Our results show that full-cycle scientific research remains challenging for current agent systems: even the strongest agents achieve limited rediscovery success (<50 F1), exhibit high variance across runs, and display recurring failure modes in experimental design, execution, and evidence-based reasoning. FIRE-Bench provides a rigorous and diagnostic framework for measuring progress toward reliable agent-driven scientific discovery.