Can LLMs Uncover Hidden Physical Laws through Free Exploration?

AI summary

Overview

Research area: AI for scientific discovery; autonomous LLM agents in simulated physical environments (natural language processing / agent evaluation). Technical level: Intermediate — readers should be comfortable with LLM agents, tool use, and physics simulation, but no specialist mathematics is required. Scope (one sentence): The paper introduces LongChang, an automated framework of MuJoCo "glass room" worlds with hidden altered physical laws, and tests whether 12 LLMs spontaneously notice anomalies, pose their own research questions, build instruments, and establish the altered laws through experiments without any assigned task.

What This Paper Is About

Scientific discovery often starts when something unexpected happens and a researcher decides it is worth investigating. Existing benchmarks tell an LLM which law to find or what anomaly to explain; this paper removes all such guidance and asks whether an LLM can notice that a simulated world behaves oddly (for example, heavier objects falling more slowly) and then design experiments on its own to pin down the hidden rule. The framework also includes a loop in which separate LLM agents use the explorers' struggles to author harder new scenes, so the whole discovery-and-design cycle runs without a human in the loop.

Key Contributions

1. LongChang, a task-free physical-law discovery framework. A base set of 15 scenes — one ordinary-physics control plus 14 scenes each altering a single law (e.g., mass-dependent fall, oscillating gravity, a speed limit of 2.2, tiles with different restitution, delayed velocity copying) — in which the evaluated LLM is an "explorer" that receives no research task and no hint that physics has been altered.
2. A full closed-loop system of separate LLM agents. One set of agents designs and reviews scenes, another reviews explorer-built apparatus, and a judge evaluates final reports against the hidden laws and the full exploration record, requiring both law correctness (C) and evidence support (S) before counting a discovery (D).
3. Scene evolution driven by exploration records. LLM designers use unresolved questions and corrected errors from earlier explorations to generate, refine, or compose new scenes, with automated tests and a reviewer gating acceptance, so the discovery cycle continues automatically.
4. An empirical study of 12 LLMs combining model comparison, controlled interventions (task guidance, 100 vs. 500 turns, apparatus access), and qualitative case studies of question formation, hypothesis discrimination, and tool construction.

Main Findings

• Nearly all models fail at free discovery. GPT-6-Astra reaches a discovery rate of 16.7%; the other 11 evaluated LLMs all score 0.0%.
• Correctness and evidence support come apart. GPT-6-Astra has an evidence support rate of 97.6% but law correctness of only 19.0%; Claude-Opus-5 shows the reverse, with law correctness of 16.7% and an evidence support rate of 0.0%.
• Partial findings dominate. Reports often establish a real effect without its conditions — e.g., a Scene 02 report supports attraction between the tested red balls but never establishes its distance dependence or applicability to other red spheres.
• Stopping early wastes budget. Gemini-3.7-Flash stops early in 100.0% of runs and averages 10.9 turns; raising the limit from 100 to 500 turns does not help, as GPT-6-Astra uses about 40 turns under either limit and discovery drops from 16.7% to 14.3%.
• Building tools helps but does not guarantee discovery. The GLM models use about 90 turns and succeed in building apparatus in every run (construction success 45), yet make no discoveries, and fail more than 60% of build attempts (construction failure 64.0% for GLM-5.3, 71.4% for GLM-5.3-Flash). Disabling apparatus drops GPT-6-Astra's discovery from 16.7% to 4.8% and law correctness from 19.0% to 4.8%.
• Persistence alone is insufficient. Grok-4.7 stops early least often (24.4%) but makes no discoveries, with a construction failure rate of 88.5%.
• Guidance raises discovery for an already-strong explorer. Giving GPT-6-Astra a law-discovery task, a warning that laws may be altered, and a scene-specific clue raises discovery from 16.7% to 28.6% and law correctness from 19.0% to 31.0%, while mean exploration length falls from 39.3 to 30.9 turns; evidence support stays at 97.6%.
• Models over-interpret ordinary physics. In the normal control, 36.1% of reports are rejected for false anomaly claims or unsupported explanations — for example, DeepSeek-Flash attributing longer travel on the gray tile to lower friction while comparing pushes from different starting speeds.
• Evolved scenes defeat the best explorer. GPT-6-Astra scores 33.3% discovery on Scene 15 and 66.7% on Scene 14, but 0.0% on all three scenes evolved from a corrected error, despite 100.0% evidence support in each.
• Qualitative behavior differs sharply. In the Scene 15 case study, GPT-6-Astra abandons its plan to chase an unexplained restart and finds that pushes or throws pause the red ball's northward force for ten counts, while Gemini-3.7-Flash sees the same restart and continues its original plan.

Methodology in Plain English

The authors build a simulated room in MuJoCo containing movable balls and boxes, some coloured floor tiles, and ordinary rigid-body physics. For each test scene, a designer LLM writes the scene and secretly changes exactly one law — say, making downward acceleration depend on mass, or making one tile bouncier than another — and records that law, its targets, triggers and constants as ground truth the explorer never sees. The explorer LLM gets a prompt saying only "explore freely and report what you learn," plus an action interface (push, throw, grasp, move, release, wait, mark) and structured observations with positions in bricks and time in counts. If the explorer wants a measurement device it cannot find, it can submit an apparatus design as MuJoCo XML; a reviewer LLM checks it, and a rejected design can be revised and resubmitted. Each turn the explorer writes a note and takes one action; at the end it must write a report citing specific turns, using only interpretations it already recorded.

A judge LLM then compares the report against the hidden law and the full record. It scores law correctness (did the report identify the law, its objects, conditions, triggers and constants identifiable in the reported units, allowing different mechanisms with identical observable predictions?) and evidence support (do the cited experiments actually justify the claims at the stated certainty?) separately. Only a report passing both counts as a discovery; a lucky guess can pass correctness alone, and a supported but incomplete finding can pass support alone. Finally, a designer LLM reads the exploration records — especially errors that later observations corrected — and generates, refines or composes scenes that preserve and complicate that same mechanism, with automated tests and a reviewer checking that the new scene actually implements the intended law before it enters the evaluation set.

Why This Matters

The paper isolates a capability that benchmarks with preassigned targets cannot measure: deciding what to investigate. It shows that current LLMs can be diligent experimenters with strong evidentiary discipline while still failing to close the loop into a stated law, and that simply granting more turns or more tools does not fix this.

Real-world applications:

• Simulation and engine debugging: agents that notice when a physics engine, game world, or digital twin deviates from expectation and localize the cause.
• Autonomous lab and instrument pipelines: agents that decide a measurement is needed, design a custom apparatus, and iterate on a failed design, as in the Scene 15 weighing-platform case study.
• Anomaly triage in engineering and operations: distinguishing a genuine regime change from a misread of ordinary noise, which the 36.1% control rejection rate shows is a live failure mode.
• Science education and assessment: evaluating whether a model, or a student using such a system, can turn an unexpected observation into a controlled experiment.

Industry relevance: developers of agentic systems, scientific ML platforms, robotics and simulation vendors, and anyone deploying LLM "researchers" receive a concrete failure taxonomy — early stopping, unresolved anomalies, unstated preconditions, partial laws — plus a task design (hidden law, no hints, dual correctness/evidence scoring) that other evaluations can adopt.

Future Directions

• Closing the identification gap: making supported-but-incomplete findings into complete laws, especially recovering conditions, triggers and timing dependencies, which the three evolved scenes show remaining unsolved at 0.0% discovery.
• Preventing premature termination: the paper shows extra turns are worthless when explorers stop early, so methods for recognizing open questions and continuing are needed.
• Extending beyond solvable worlds: the authors note that LongChang's altered laws have known answers, and that applying the framework to genuinely open scientific questions requires new ways of evaluating exploration and its findings.
• Trustworthy automated judging: the authors flag as a risk that LLM judges may endorse unsupported findings and mislead later research, and that any physical realization of generated experiments or apparatus would need independent validation and expert safety review before use.

Target Audience

Researchers in AI for science and LLM-agent evaluation, benchmark designers, and engineers building autonomous experimentation or simulation-debugging systems. It is also useful for cognitive-science and science-education readers interested in machine hypothesis formation, and for practitioners who want a concrete, runnable test of whether an agent can notice the unexpected rather than merely execute a stated task.

Authors’ abstract

Scientific discovery often begins with an unexpected observation that prompts a new research question. In a world with altered physical laws, can large language models (LLMs) notice unexpected behavior, such as heavier objects falling more slowly, and discover the law through experiments without being told what to investigate? To evaluate this ability, we introduce LongChang, an automated framework that creates simulated worlds with altered physics, where one or more laws differ from ordinary physics. The evaluated LLMs act as explorers, manipulating objects and building tools to investigate their own questions without research tasks or hints about these laws. Separate LLM agents generate and review scenes, review tool designs, and evaluate reports for correctness and experimental support using the altered physical laws and exploration records. These records guide scene evolution for further exploration, allowing the full cycle to run without human intervention. Among 12 LLMs, GPT-6-Astra achieves a discovery rate of 16.7%, while the other 11 score 0%. The exploration records reveal uninvestigated anomalies and only partial identification of the altered physical laws.