From Research Goals to Scientific Closure: Evaluating Agents Through Evidence-Supported Physics Research

AI summary

Overview

Research area: AI agents for autonomous scientific research (benchmarking and evaluation), applied to computational physics — strong-field physics, attosecond science, solid-state/condensed-matter physics, quantum optics, and relativistic quantum dynamics.

Technical level: Advanced. The paper assumes familiarity with LLM agent evaluation, rubric-based grading, and research-level physics concepts (attoclock, screening length, Coulomb potentials), though its central argument — that producing correct results is not the same as resolving a research question — is accessible to non-physicists.

Scope: The paper introduces PhysClosureBench, a 50-task expert-verified physics benchmark with a two-level evaluation scheme (Atomic Score and Research Closure RC0–RC4), and evaluates seven frontier language-model agents over 1,050 scored attempts.

What This Paper Is About

AI agents can now write code, reason, and produce plausible scientific results, but producing a result is not the same as carrying an investigation to a defensible conclusion. The authors ask whether agents can independently organize an investigation so that the evidence, interpretation, and validity conditions required to answer a specified research question are actually established — a property they call scientific closure. To measure it, they build a physics benchmark where the agent is given a goal and constraints but no solution recipe, and they score both how much local scientific work was done and whether the investigation as a whole closed.

Key Contributions

1. PhysClosureBench — a benchmark of 50 expert-verified, goal-specified physics research tasks derived from published studies in journals such as Physical Review Letters and Physical Review A, spanning strong-field atomic and molecular physics, attosecond science, solid-state and condensed-matter physics, quantum optics, and neighboring quantum light–matter regimes. Each task fixes the objective, physical setting, parameters, constraints, and computational resources, but leaves the investigation strategy open.

2. A two-level evaluation framework. The Atomic Score sums 20 predefined task-specific criteria worth 0.5 points each (0–10), grouped into five dimensions: Formulation (F), Output (O), Identification (I), Mechanism (M), and Validation/Scope (V). Research Closure (RC0–RC4) separately judges whether the investigation satisfies the task-specific evidence requirements and scope conditions needed to resolve the research question — deliberately not a threshold on the Atomic Score.

3. An empirical study of seven frontier models over all 50 tasks with three independent runs per model–task pair (1,050 final scored attempts), reporting Atomic Score, functional completion by dimension, RC distributions, and run-to-run consistency.

4. A human consistency check of the automated grader. Five physics-trained experts reviewed 100 evaluation cards covering all 50 tasks, producing 200 selected criterion-level judgments; automated grading agreed in 81.8% of atomic-criterion judgments and 75.9% of judgments on criteria explicitly linked to Research Closure.

Main Findings

• Best model still closes rarely. Claude-Opus-5 has the highest mean Atomic Score (6.29/10) and reaches RC4 in only 32.7% of attempts. GPT-5.6-Sol follows at 5.83/10 (RC4 22.7%) and Kimi-K3 at 5.17/10 (RC4 19.3%); the remaining models score 4.50 or below. Gemini-3.1-Pro is lowest at 2.85/10 with an RC4 rate of 0.7%.

• Setup is easier than interpretation. Formulation has the highest completion rate among F/O/I/M for every model; for Claude-Opus-5 it reaches 85.6%, versus 60.1% in Output, 54.6% in Identification, and 52.8% in Mechanism. Identification and Mechanism remain hard across the board — even the strongest model completes only about half of applicable criteria in each.

• High scores do not imply closure. Among the 229 attempts scoring at least 8/10, 83 (36.2%) fall short of RC4; 55 of those remain at RC2, meaning a requirement for the main scientific conclusion itself is unresolved, not merely validation.

• The residual gap concentrates in Identification and Mechanism. Among high-scoring attempts below RC4, 16.0% of Identification criteria and 20.8% of Mechanism criteria are unmet, versus only 2.1% and 3.0% among RC4 attempts (RC4 n=146, below-RC4 n=83).

• Validation is the weakest scope-related element. Across the benchmark, scope justification is satisfied 86.9% of the time, compared with 59.6% for required validation and 47.8% for risk-linked validation.

• A higher RC4 rate can hide widespread failure. DeepSeek-V4.1-Flash reaches RC4 in 10.7% of attempts — more often than the three lower-scoring models — yet 50.0% of its attempts remain at RC0, meaning the essential scientific basis was never established.

• Closure is demonstrated more often than it is reliable. Claude-Opus-5 reaches RC4 on at least one of three runs for 50.0% of tasks but on all three runs for only 18.0%. GPT-5.6-Sol and Kimi-K3 each reach RC4 at least once on 40.0% of tasks, but in all three runs on only 10.0% and 4.0% respectively. DeepSeek-V4.1-Flash reaches RC4 at least once on 24.0% of tasks and on none in all three runs.

• Failures come in qualitatively different kinds. In the Case 30 attoclock comparison, Gemini-3.1-Pro generates relevant outputs but misdescribes increasing or turning-over screened responses as monotonically decreasing (RC2); GPT-5.6-Sol explains the response changes but leaves unresolved a high-field applicability risk exposed by its own Coulomb barrier diagnostic (RC3); Kimi-K3 corrects an angle-sign inconsistency, explains the exit-radius/screening-length competition, and justifies applicability limits (RC4).

• The grader is mostly but not fully aligned with experts. Automated grading matched expert judgment on 81.8% of 200 selected atomic-criterion judgments and 75.9% of RC-relevant judgments (60 agreements among 79 binary judgments, drawn from 84 judgments across 72 cards and 44 tasks).

Methodology in Plain English

The authors built tasks by taking published physics studies and turning each into a bounded research assignment: here is the objective, the physical setting, the parameters, and the compute budget — now figure out how to answer it. Every task was manually reproduced before being admitted, which both confirmed it was solvable and calibrated the resource limits (for example, one case allows 96 CPU cores and 720 minutes, others 8 cores and 30 minutes).

Agents ran in isolated, network-disabled containers through OpenCode (version 1.18.17), with Slurm allocating the task-specific cores and wall time. They could write and run code, generate data and figures, and manage their own files, but could not search the web or ask for help; they received a common instruction file and a task directory, and had to place final materials under a submission directory. Incomplete work was still scored: timeouts were graded on the science available at that moment without an extra timeout penalty, and attempts with no assessable result received zero atomic credit and RC0. Infrastructure-invalid runs were excluded and rerun.

Grading used GPT-5.6-Sol at high reasoning effort in a staged protocol: an independent first-pass grade against the case rubric, a targeted review with access to that first pass, and adjudication only when a residual disagreement could change the Atomic Score, Research Closure, or validity. Each case's 20 half-point atoms were distributed across dimensions with Mechanism weighted heaviest — across all 1,000 rubric atoms, M accounts for 369 (36.9% mean score weight), O 192 (19.2%), F 164 (16.4%), I 163 (16.3%), and V 112 (11.2%). Criteria that do not apply are marked N/A and keep their 0.5-point contribution but are dropped from both numerator and denominator of the completion rate. Closure level is assigned from task-specific requirements and their prerequisites, not from a score cutoff. Analyses treat each attempt as the unit of observation, with 150 attempts per model.

Why This Matters

Impact on research: The paper argues that benchmark scores measuring answer correctness, executable artifacts, or replication miss a distinct capability — whether an investigation actually establishes a defensible answer. It gives the field a measurable construct (Research Closure) and shows that local progress and overall closure can diverge sharply, which matters for anyone claiming agents can "do science."

Real-world applications:

• AI-assisted physics and simulation workflows, where the failure mode of "computed the right numbers, drew the wrong conclusion" is precisely what the benchmark isolates.
• Evaluation design for scientific agents — the two-level rubric (fine-grained atoms plus an overall closure state) is portable to chemistry, materials, or biology task suites.
• Reliability auditing of autonomous research tools, using the paper's at-least-once versus all-three-runs comparison to separate capability from consistency before deployment.
• Reviewer and journal-support tooling, where flagging unresolved applicability limits and unsupported mechanisms is the useful signal rather than a plausible-looking result.

Industry relevance: Labs and companies deploying agents for R&D need to know not just whether a model can produce a result but whether it knows when a conclusion is licensed by its evidence. The finding that risk-linked validation is satisfied only 47.8% of the time, and that models often leave their own diagnostic warnings unresolved, is a direct argument for validation gates in agent pipelines.

Future Directions

• Closing the Identification and Mechanism gap. These dimensions are the dominant residual failure among high-scoring attempts (16.0% and 20.8% unmet) while Formulation and Output are nearly saturated — the open question is what training or scaffolding makes an agent connect generated evidence to a physical explanation.

• Making closure reliable rather than occasional. The gap between RC4 at least once (50.0% of tasks for the best model) and RC4 in all three runs (18.0%) is unexplained by the paper; understanding run-to-run variance and whether it comes from strategy choice, time allocation, or interpretation errors is a natural next study.

• Handling applicability limits explicitly. Since unresolved applicability concerns produce RC3 outcomes even when the mechanism is right, agents need mechanisms for restricting conclusions to evidence-supported regimes — a capability the benchmark currently only diagnoses.

• Extending beyond physics and strengthening grading. The authors position closure as a general property of investigations; whether the rubric transfers to other domains, and how to raise grader–expert agreement above the reported 81.8% atomic and 75.9% RC-relevant rates, remain open.

Target Audience

Researchers building and evaluating scientific AI agents; benchmark designers who need to distinguish task-level correctness from investigation-level validity; physicists and domain experts interested in where LLM agents still fail on research-grade problems; and research-engineering teams in industry who need to assess whether an autonomous agent's conclusions are trustworthy before acting on them. Readers without a physics background can follow the framing and results, but the case analysis and task design assume comfort with research-level physics.

Authors’ abstract

Can AI agents organize scientific investigations that lead to well-supported conclusions? To evaluate this capability, we introduce PhysClosureBench, a benchmark of 50 expert-verified, goal-specified research tasks spanning strong-field, attosecond, and neighboring physics areas. Each task specifies the research objective and essential scientific constraints, while leaving the agent to organize the investigation and determine how to generate and validate the evidence needed to answer the question. A 20-criterion Atomic Score measures local scientific accomplishment, while Research Closure (RC0--RC4) assesses how fully the investigation resolves the specified research question based on whether its indispensable evidence requirements and applicable scope conditions are satisfied. We evaluate seven frontier LLMs, where Claude-Opus-5 achieves the highest mean Atomic Score (6.29/10) but reaches RC4 in only 32.7% of attempts. Across the benchmark, 36.2% of attempts scoring at least 8/10 still fall short of RC4, while all seven models reach RC4 on substantially fewer tasks in all three runs than in at least one. Case-level analysis further illustrates how misinterpretation of generated results and unresolved physical applicability limits can lead to incomplete closure. Together, these results reveal a persistent gap between AI agents' local scientific accomplishments and their ability to achieve complete research closure.