FrontierPhysics: Evaluating Agents for End-to-End Physics Research

AI summary

Overview

Research area: AI agent evaluation / AI for science — specifically benchmarking autonomous LLM agents on end-to-end physics research.

Technical level: Intermediate. The paper is readable without specialist physics knowledge, but it assumes familiarity with LLM agents, tool use, benchmarking, and rubric-based grading.

Scope (one sentence): The paper introduces FrontierPhysics, a 56-task benchmark authored by 42 practicing physicists from their own research projects, and reports that 22 frontier models across 26 configurations all fail to pass more than 46 of 168 rollouts, with failures driven mainly by weak rigorous reasoning and missing expert intuition rather than gaps in physics knowledge.

What This Paper Is About

Existing physics benchmarks test fixed questions, isolated sub-tasks, or paper reproduction where the original publication is handed to the agent — so the research question and solution path are largely given away, and the tasks saturate quickly. FrontierPhysics instead gives an agent only a high-level prompt of the kind a principal investigator would give at the start of a project, and requires it to review the literature, plan, run experiments or computations, produce verifiable numerical outputs, and write a manuscript. The goal is to measure whether agent research is both correct and publishable, not just whether a final answer matches.

Key Contributions

1. A benchmark built from real research, not textbook problems. 56 tasks were accepted from 182 initial submissions, written by 42 physics researchers from 26 institutions across nine domains (Condensed Matter, AMO, Quantum Information, Astrophysics, Fluid Dynamics, Statistical Physics, Photonics & Imaging, High Energy Physics, Chemical Physics). For 18 tasks the underlying research has been published, including four based on papers in Nature, with others in Nature Communications, Physical Review Letters, and Physical Review X. Contributors report roughly 270 days per task on the original projects, totaling more than 8,000 hours of research.

2. A three-stage quality pipeline designed to suppress false negatives and false positives. Every task passes a general quality review (provenance checks, a requirement of at least two weeks of original research effort, rejection of purely AI-generated content), a domain peer review by researchers (47% Ph.D. students, 37% Ph.D. holders), and a failure-mode analysis based on at least three local runs with state-of-the-art agents. A six-person admin team of physics Ph.D. researchers gives final approval.

3. A dual grading scheme combining deterministic verifiers with expert rubrics. Test scripts check numerical artifacts against withheld reference values; a reviewer agent grades the trajectory and paper.pdf against an author-written rubric split into binary blockers (scientific validity gates) and weighted non-blockers scored 0/1/2 with integer weights from 1 to 10. A submission scores zero unless the verifier passes and every blocker passes, so strong writing cannot compensate for wrong results, and correct results cannot compensate for an unpublishable manuscript.

4. A manually validated failure-mode taxonomy. Domain experts audited all 168 GPT-6 Astra rollouts and labeled them, producing a taxonomy (PHY, SP, RRMIS, LIT, INS, REAL, INT, COH, RIG, FALSE, MIS, EXE, HAL, HACK) that identifies what actually breaks frontier agents.

Main Findings

• Every tested configuration fails most of the time. Across 22 models in 26 configurations and 4,364 rollouts, no configuration passes more than 46 of its 168 rollouts. Overall pass rates are below 30% for all agents tested; Claude Opus 5.5 in Claude Code CLI has the highest pass rate at 27.4%. Each model attempted all 56 tasks three times at its highest available inference effort, with four GPT models also run at xhigh.

• The dominant failure is reasoning discipline, not physics knowledge. Among the 168 GPT-6 Astra rollouts, lack of rigorous thinking accounts for 48% of failures (e.g., circular self-verification, where the agent's validation method depends on its own wrong assumption, and unhandled edge cases), missing expert intuition for 34%, literature synthesis for 27%, and instruction following for 26%.

• Agents miss the "expert instinct" parts of research. Typical cases include failing to identify the dominant error source in an experiment — not from ignorance of the physics, but from lacking the intuition about what matters — and continuing to tune the wrong knob while debugging.

• Nine of 168 rollouts were incorrectly graded as failures. Eight FALSE labels (correct work failed by unclear instructions, contradictory verifiers, or overly strict tolerances) and one reviewer-agent misgrade mean the true GPT-6 Astra pass rate is 20.2% (34/168) rather than the recorded 14.9% (25/168). Only the GPT-6 Astra rollouts could be reliably corrected this way, because manually reading trajectories involving subagents is difficult for other configurations.

• The rubric grades human expert work highly. For the 33 tasks where authors supplied their own papers or manuscripts as references, those materials score 96.2% on average under the corresponding rubrics, indicating the rubrics are aligned with genuine expert output.

• The reviewer agent is largely stable but has a family bias. Regrading the first-round rollouts with Claude Opus 5.5 in Claude Code CLI instead of GPT-5.6 Sol (xhigh) in Codex CLI gives the same pass/fail decision for almost all GPT-6 Astra rollouts, but the Opus 5.5 reviewer is more lenient on rollouts from its own model family.

• Cost buys some performance, with exceptions. Pass rate generally rises with mean inference cost per rollout, and GPT models are generally more cost-efficient. For some Claude models, however, pass rate decreases as cost increases, in the order Opus 5.5, Fable 5.1, and Opus 5.

• Domain-level differences are not yet trustworthy. The paper reports per-domain pass rates for Opus 5.5 and GPT-6 Astra but notes that, with few tasks per domain, the differences between domains are uncertain. Fifteen tasks also carry secondary domain labels that do not add extra observations.

Methodology in Plain English

The researchers did not write the tasks themselves as benchmark designers; they recruited working physicists — Ph.D. students, postdocs, research scientists, averaging 420 citations each — to convert research projects they had personally completed into agent tasks. Each task starts with a one-page proposal-like brief that states the physics and the goal and poses open questions, but withholds explicit methodology and solution paths. Supporting data and files are placed in the environment so the agent can configure and run the relevant computational or experimental workflow. The agent must submit both verifiable artifacts (computed quantities, simulation outputs, processed data) and a paper.pdf.

Review happens before and after deployment. Before acceptance, contributors must prove provenance (reference papers or patents, project timeline, time spent), and a domain expert checks that the prompt does not over-specify the problem, that the rubric encodes real physics insight, and that the initial failure-mode traces do not reveal unjustified tolerances or ambiguous instructions. After acceptance, each rollout is run in a cloud sandbox with a uniform 10-hour timeout (Daytona for tasks under 4 vCPUs, 16GB RAM and 10GB storage; Docker in Azure VMs with Lambda API access for heavier tasks). The same reviewer agent, GPT-5.6 Sol (xhigh) in Codex CLI, grades every rollout in a separate sandbox, with access to the trajectory and output files rather than the paper alone. Experiments and trajectory/verifier/rubric tracking run through BenchFlow CLI.

Finally, three contributors per task wrote failure-mode reports on three GPT-6 Astra rollouts each; the admin team re-checked these against the actual trajectories and output files, consolidated recurring failures into a taxonomy, and manually labeled all 168 rollouts. Infrastructure-caused endings (lost sandboxes, provider errors surviving retries) were re-run; model-caused endings were kept in the denominator.

Why This Matters

Impact on research. The paper argues that the binding constraint on AI research agents is not domain knowledge but scientific judgment — knowing when your own verification is circular, which error source dominates, and which paper actually matters. That reframes what to train and measure. It also contributes a methodological lesson for benchmark builders: nine of 168 rollouts were graded wrongly, echoing recent work where expert regrading finds many benchmark failures are grader errors, incorrect reference solutions, or underspecified questions. FrontierPhysics is, per its own comparison table, the only surveyed benchmark that combines tasks from contributors' own research, no reference paper given, graded literature review, graded manuscript, expert rubrics, deterministic verifiers, and manual failure analysis.

Real-world applications (as motivated by the paper's framing):

• Evaluating and selecting AI research assistants for physics labs and computational groups, where the output must be a defensible manuscript plus reproducible numbers rather than a chat answer.
• Automating literature review, hypothesis generation, code-based experimentation, and scientific-software optimization — capabilities the paper cites as already demonstrated in limited settings.
• Experimental and instrument-facing work in the task set, such as ion-trap design, Rydberg apparatus behavior, and detector/spectrometer pipelines, which points toward agent assistance in experiment planning and error-budget analysis.
• Benchmarking and regression-testing agent products before deployment in scientific workflows, using the blocker/verifier gates as an acceptance criterion.

Industry relevance. AI labs building research or "deep research" agents get a hard, unsaturated evaluation set with a documented failure taxonomy to target. Cloud and infrastructure providers are implicated directly, since rollouts depend on parallel cloud sandboxes, GPU access, and long timeouts. The cost-versus-pass-rate curves matter commercially: they show spending more inference compute does not reliably buy scientific reliability, and in some Claude models corresponds to lower pass rates.

Future Directions

• Extend the failure analysis beyond a single model–harness pair. The authors state explicitly that the current taxonomy covers only GPT-6 Astra, because manually reading trajectories with subagents involved is hard, and that other models or model–harness pairs may fail in ways outside this taxonomy. It is also why only GPT-6 Astra's false negatives could be corrected.

• Reduce false negatives at scale. Detecting the nine misgraded rollouts required both domain expertise and time. Better tooling and protocols for finding grader errors, bad reference values, and ambiguous instructions remain an open problem.

• Move toward more open-ended tasks and more flexible scoring. The conclusion calls for tasks and evaluation frameworks that can credit exploratory reasoning, partial progress, and unexpected findings, rather than a binary pass against a withheld answer — the direction needed if agents are to go from specified tasks to setting their own problem definitions.

• Harden reviewer agents. The observed leniency of Claude Opus 5.5 toward its own model family indicates that agent-as-judge grading needs bias controls if it is to be used as a primary evaluation instrument.

Target Audience

Researchers and engineers building or evaluating AI agents for scientific work; benchmark designers who need a model for expert-authored, rubric-plus-verifier evaluation with manual failure-mode auditing; AI lab and platform teams deciding how to measure research-agent capability; and physicists curious about where current agents actually break down on real projects. Readers seeking per-model leaderboard numbers beyond the top-line figures should consult Table 2 and Appendix D, which this summary does not reproduce in full.

Authors’ abstract

AI agents are playing an increasingly important role in physics research. However, existing physics benchmarks only evaluate fixed questions, isolated sub-tasks, or paper reproduction with given solutions, and rarely check research quality alongside correctness. We introduce FrontierPhysics, an end-to-end benchmark with 56 tasks made by 42 physicists based on their own research, covering nine domains across theoretical, computational, and experimental physics. Each task requires agents to review the literature, develop a research plan, conduct research and experiments, and produce structured outputs and scientific manuscripts. Combining deterministic verifiers with expert-authored rubrics, FrontierPhysics evaluates whether agents' research is correct and publishable. Across 22 models in 26 configurations, no configuration passes more than 46 of its 168 rollouts. Manual review of agent trajectories shows that frontier agents' main weaknesses are insufficiently rigorous reasoning and limited expert intuition, rather than gaps in physics knowledge or execution errors.