NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agents

AI summary

Overview

Research area: AI-driven scientific discovery, specifically benchmarking LLM agents on scientific law discovery. Technical level: Advanced. Scope: The paper introduces NewtonBench, a benchmark of 324 interactive scientific law discovery tasks across 12 physics domains, and reports an evaluation of 11 state-of-the-art LLMs on it.

What This Paper Is About

Existing benchmarks for LLM-driven scientific law discovery force a trade-off between scientific relevance, scalability, and resistance to memorization, and they treat discovery as static function fitting on tabular data rather than the interactive experimentation that characterizes real science. NewtonBench is built to resolve both problems: it generates novel, physics-grounded laws via systematic "counterfactual law shifts," and it requires agents to actively probe simulated model systems to uncover a hidden target equation.

Key Contributions

1. A counterfactual law shift method that defuses the benchmark trilemma. By systematically mutating the mathematical structure of 12 canonical physical laws (altering operators or exponents, with dimensional units of embedded constants adjusted to preserve physical coherence), the authors generated 108 shifted laws that are simultaneously scalable, scientifically relevant, and memorization-resistant.

2. A shift from static function fitting to interactive model discovery. Instead of fitting a formula to a provided table, agents must call a <run_experiment> tool, choose input values for a simulated system, and interpret outputs. The target law is embedded in a model of up to three hierarchical configurations: Vanilla Equation (target law alone), Simple System, and Complex System, where assisting equations are given and confounding variables must be disentangled.

3. Two independent axes of difficulty control. Intrinsic difficulty comes from the cumulative number of mutations (Easy: 1–2 mutations from the canonical law; Medium: 1–2 further mutations from Easy; Hard: 1–2 further mutations from Medium), while extrinsic difficulty comes from the surrounding system complexity. Each of the 108 shifted laws is instantiated in three systems, giving 324 total tasks.

4. An optional code assistance setting and an evaluation protocol. Agents may optionally invoke a <python> code interpreter, shifting the evaluation from compute-bound to discovery-bound. Performance is measured by Symbolic Accuracy (binary, constant-insensitive equivalence) and Root Mean Squared Logarithmic Error (RMSLE), with an LLM-as-a-judge equivalence check reported at 98.3% agreement with human experts.

Main Findings

• NewtonBench is reasoning-intensive and fails non-thinking models. The three non-thinking LLMs (GPT-4.1-mini, GPT-4.1, DeepSeek-V3) all achieve overall symbolic accuracies below 10%, showing limited proficiency (20-40% accuracy) even in the simplest setting.

• Frontier models lead, but the gap widens with complexity. Most reasoning models achieve overall accuracies between 30-80%. GPT-5 reaches 75.9% average symbolic accuracy (RMSLE 0.2490, 19.15k tokens) and Gemini-2.5-pro reaches 65.4% (RMSLE 0.3535, 21.54k tokens). In the most challenging setting (Complex System, Hard), GPT-5 and Gemini-2.5-pro retain 40.3% and 16.7%, while all other reasoning models fall below 6%.

• Code assistance has a dichotomous effect. For less capable models (symbolic accuracy below 40%), the code interpreter substantially improves accuracy; for more capable models (symbolic accuracy at or above 40%), it slightly degrades performance. GPT-5 drops from 75.9% to 73.3% and Gemini-2.5-pro from 65.4% to 63.7% with code assistance.

• The degradation is explained by premature exploitation. The performance divergence is most pronounced when the code budget increases from zero to one. Analysis of exploration-signature tokens (e.g., "What if", "Alternatively") and exploitation-signature tokens (e.g., "Confirm", "Verify") shows a sharp drop in exploration rate for Gemini-2.5-flash as the code budget rises from zero to one, while the rate stays stable for Qwen-3-235b. GPT-5-mini allocates a much smaller proportion of its code use to basic calculation than GPT-4.1, favoring function-fitting instead, which can cause it to satisfice on a local optimum.

• Symbolic accuracy is extremely fragile to observational noise. Testing GPT-5-mini with four Gaussian noise levels (0.0001, 0.001, 0.01, 0.1), the introduction of just a 0.0001 noise level caused a 12-16% reduction in accuracy versus the noise-free setting. As noise increased from 0.0001 to 0.1, symbolic accuracy declined proportionally while RMSLE remained relatively stable. Code assistance did not affect noise robustness.

• Domain matters, and its effect grows with system complexity. Average accuracy across physics domains ranges from 18% to 54%. Bose-Einstein Distribution, described as the most advanced and obscure domain in the benchmark, yields the lowest average at 18.1%. Heat Transfer yields 68% in the simple setting (comparable to Acoustic Velocity at 60%), but plummets to 3.3% in the complex setting while Acoustic Velocity remains at 45.0%.

• Reasoning models scale inference with difficulty; non-reasoning models do not. Compared against non-reasoning LLMs (GPT-4.1/4.1-mini, DeepSeek-V3), strong reasoning LLMs (Gemini-2.5-pro/flash, GPT-5/5-mini) substantially increase token consumption as task difficulty rises, whereas token cost for non-reasoning models remains consistently low even when they consume more experiment rounds.

• NewtonBench is far larger than prior law-discovery benchmarks. For comparison, the paper lists AI Feynman at 120 problems, EmpiricalBench at 9, LLM-SR at 4, EvoSLD at 5, LSR-Transform at 111, LSR-Synth at 128, and PhysSymbol at 5,000, several of which are marked as not fully memorization-free, not model-system-oriented, or relying on passive observation. NewtonBench is listed at 324 Model System problems with active exploration.

Methodology in Plain English

The authors started from 12 canonical physical laws chosen because they are foundational, span diverse physics domains, and can be mutated in a dimensionally consistent way. They then applied counterfactual law shifts — analogous to counterfactual reasoning in physics — implemented as mutation operations on each law's expression tree, changing either operators (for example, addition to multiplication) or numeric constants (for example, turning a quadratic relationship into a cubic one). Because such edits can break dimensional consistency, every target law includes at least one physical constant whose units are systematically adjusted to compensate. Three domain-expert co-authors curated the 108 shifted laws under strict plausibility guidelines.

Each shifted law was then embedded in three virtual systems of increasing complexity. In the Vanilla Equation setting the model contains only the target law. In Simple System and Complex System settings the target law sits inside a larger ordered sequence of equations with confounding variables, and the other equations are disclosed as known assisting equations so the agent can isolate the target.

Agents interact with the environment by proposing input values for the system and receiving the resulting outputs, iteratively building their own dataset. In the code assistance setting they can also write and execute Python. Answers are scored on structural equivalence to the ground-truth law (with constant values ignored) and on RMSLE, chosen over NMSE for numerical stability across physical quantities spanning multiple orders of magnitude. Eleven LLMs were evaluated, including both non-reasoning and reasoning models, with results aggregated over 12 domains by equation difficulty and system complexity, reported as mean and standard deviation from four runs. All evaluations ran through public APIs (OpenRouter and OpenAI-API), with total experimental cost estimated at 10,000 USD.

Why This Matters

NewtonBench provides a scalable, memorization-resistant, scientifically grounded testbed for measuring whether AI agents can do genuine scientific discovery rather than recall or curve-fit. It also surfaces a counterintuitive failure mode of tool use — a code interpreter helping weak models but hurting strong ones — which has direct implications for how agentic systems should be designed and how tools should be budgeted.

Real-world applications:

• Automated experiment design. The interactive, black-box experimental protocol mirrors how an agent might plan and select conditions in a real lab or simulation pipeline rather than passively analyzing a fixed dataset.
• Materials and drug discovery pipelines. Agents that can isolate a target relationship inside a system with confounding variables are directly applicable to screening problems where many coupled factors obscure the quantity of interest.
• Physics and engineering modeling. The benchmark's framing of recovering equations for physical quantities tests the kind of symbolic abstraction used in deriving transfer laws and conduction relations, among the covered domains.
• Evaluation infrastructure for AI-for-science claims. Teams building or marketing discovery-capable models can use the benchmark's difficulty axes and noise sensitivity tests as a litmus test, as the authors position it.

The paper also argues its findings generalize beyond physics. The ethics statement notes the physics domain was chosen as the most representative ground for scientific law discovery but that the results should extend to chemistry and biology, since isolating variables to derive mathematical relationships is a similar cognitive process whether one studies reaction kinetics or enzyme dynamics.

Industry relevance: the fragility results and the exploration-exploitation trade-off matter to anyone deploying LLM agents in research or engineering loops, especially where observational noise is unavoidable. The authors' aim of pushing evaluation from compute-bound to discovery-bound is a direct statement about where frontier model development effort should go. The work was supported by the ITSP Platform Research Project (ITS/189/23FP) from ITC of Hong Kong, the AoE (AoE/E-601/24-N) and GRF (16205322) from RGC of Hong Kong, and NVIDIA AI Technology Center (NVAITC), with two authors affiliated with NVIDIA.

Future Directions

• Mitigating premature exploitation. The authors identify managing the exploration-exploitation trade-off in agentic systems as important, particularly in how tools are leveraged by models of varying capability. Designing scaffolds or training signals that prevent premature convergence to a "good enough" law is an open problem.
• Improving noise robustness. Symbolic accuracy collapses under even a 0.0001 noise level while RMSLE stays stable. Closing this gap — deducing correct structure, not just predictive fit, from noisy data — is unresolved.
• Scaling to harder and more complex systems. Performance falls below 6% for nearly all reasoning models on Complex System, Hard tasks. Whether this barrier is one of reasoning capability, experimental strategy, or long-horizon interaction is left open.
• Extending beyond physics. The paper's ethics statement posits generalization to chemistry and biology, but does not evaluate it. Building analogous counterfactual-shift benchmarks in those domains would test that claim.
• Scope of comparison. The authors explicitly exclude traditional symbolic regression methods and LLM-based symbolic regression pipelines whose prior workflows are incompatible with the interactive protocol, so comparative placement against those approaches is not established here.

Target Audience

Researchers in AI for science and automated scientific discovery, benchmark designers evaluating reasoning and agentic capabilities, and developers building tool-using LLM agents. It is also relevant to physicists and domain scientists interested in whether LLMs can recover governing equations, and to industry teams assessing frontier models for research automation. The paper is written for readers comfortable with LLM evaluation methodology, symbolic regression terminology, and agent-environment interaction, so some familiarity with those areas is helpful despite the mostly plain-language framing of the task.

Authors’ abstract

Large language models are emerging as powerful tools for scientific law discovery, a foundational challenge in AI-driven science. However, existing benchmarks for this task suffer from a fundamental methodological trilemma, forcing a trade-off between scientific relevance, scalability, and resistance to memorization. Furthermore, they oversimplify discovery as static function fitting, failing to capture the authentic scientific process of uncovering embedded laws through the interactive exploration of complex model systems. To address these critical gaps, we introduce NewtonBench, a benchmark comprising 324 scientific law discovery tasks across 12 physics domains. Our design mitigates the evaluation trilemma by using counterfactual law shifts - systematic alterations of canonical laws - to generate a vast suite of problems that are scalable, scientifically relevant, and memorization-resistant. Moreover, we elevate the evaluation from static function fitting to interactive model discovery, requiring agents to experimentally probe simulated complex systems to uncover hidden principles. Our extensive experiment reveals a clear but fragile capability for discovery in frontier LLMs: this ability degrades precipitously with increasing system complexity and exhibits extreme sensitivity to observational noise. Notably, we uncover a paradoxical effect of tool assistance: providing a code interpreter can hinder more capable models by inducing a premature shift from exploration to exploitation, causing them to satisfice on suboptimal solutions. These results demonstrate that robust, generalizable discovery in complex, interactive environments remains the core challenge. By providing a scalable, robust, and scientifically authentic testbed, NewtonBench offers a crucial tool for measuring true progress and guiding the development of next-generation AI agents capable of genuine scientific discovery.