MOT-SR: Multi-Objective Tool-Augmented Scientific Equation Discovery with Large Language Models

AI summary

Overview

• Research area: Symbolic regression (SR) and automated scientific equation discovery, combining large language models (LLMs), external data-analysis tools, and multi-objective optimization. The paper also includes a case study in gravitational-wave astrophysics.
• Technical level: Intermediate. Readers should be comfortable with symbolic regression, Pareto-based multi-objective optimization, and general LLM prompting concepts.
• Scope: The paper proposes MOT-SR, a closed-loop framework in which two cooperating LLMs use analytical tools and a multi-objective (accuracy, generalization, complexity) Pareto front to discover compact equations, evaluated on 40 standard benchmark tasks and on extreme mass-ratio inspiral (EMRI) orbital modeling.

What This Paper Is About

Symbolic regression seeks compact, interpretable equations that fit observed data, but it is an NP-hard search problem. Recent LLM-based SR methods tend to lack systematic analysis of how variables relate to one another, and they usually optimize only fitting error — ignoring equation complexity and generalization — which leads to overfitting and premature convergence. MOT-SR addresses both gaps by having LLMs autonomously invoke analytical tools to extract structural priors, while maintaining a dynamic Pareto front over three objectives: in-domain accuracy, out-of-domain generalization, and structural complexity.

Key Contributions

1. Tool-augmented variable analysis. MOT-SR lets the LLM autonomously select and coordinate a toolbox of analytical methods spanning linear correlation (Pearson correlation, simple linear regression, residual variance analysis, PCA-based explained variance), nonlinear dependency (Spearman rank correlation, mutual information, mutual information regression), time-frequency analysis (Fast Fourier Transform, wavelet transform energy spectrum), causal inference (Granger causality, Convergent Cross Mapping), dynamic complexity (Lyapunov exponent, correlation dimension, Dynamic Time Warping), and distribution consistency (Kolmogorov–Smirnov test). The paper states that the novelty is not the tools themselves but enabling LLMs to invoke and coordinate them to distill variable dependency patterns into generation guidance.

2. Multi-objective evaluation with a dynamic Pareto front. Instead of optimizing a single fitting-error objective, MOT-SR jointly evaluates candidates on NMSE over a training-derived in-domain subset (D_train^ID), NMSE over the complementary out-of-domain subset (D_train^OOD), and equation complexity measured as abstract syntax tree (AST) length. Non-dominated sorting and pruning maintain the front across iterations, with upper-bound NMSE thresholds filtering out overly simple candidates.

3. A cooperative two-LLM evolution loop. A Meta Strategy Generator (π_stg) analyzes residual patterns of Pareto-optimal candidates and outputs a strategy composed of variable-relationship guidance (R_varrel) and structural guidance (R_struct, from commonality extraction, disparity analysis, and blind spot discovery). An Equation Generator (π_eq) then synthesizes new candidates, using a structure-diversity–guided parent sampling module based on a subtree-overlap metric.

4. Benchmark and domain validation. The framework is evaluated on 40 standard tasks (four LLM-SR benchmarks plus 36 LSR-Synth–Chemistry tasks) against classical and LLM-based baselines, and is presented as the first application of symbolic regression to EMRI orbital-evolution modeling, where it produces an interpretable correction with the lowest trajectory-level integration error on held-out configurations.

Main Findings

• Best overall accuracy on four benchmarks: On Oscillation 1 with LLaMA-3.1, MOT-SR reaches NMSE 1.27e-15 (Acc avg-0.001 of 100.00%) versus LLM-SR (Llama-3.1) at 2.55e-5 (12.67%). With GPT-4o-mini, MOT-SR reaches 1.42e-13 (99.99%). On Oscillation 2, MOT-SR (Llama-3.1) reaches 1.70e-10 with 99.45% accuracy and MOT-SR (4o-mini) reaches 4.25e-10 with 99.57%. On E. coli growth, MOT-SR reaches 6.60% / 0.0208 (Llama-3.1) and 6.32% / 0.0178 (4o-mini). On Stress-Strain, MOT-SR reaches 85.02% / 0.0150 (Llama-3.1) and 86.33% / 0.0144 (4o-mini). The paper reports that MOT-SR attains over 90% accuracy on several datasets.

• Best results on chemistry: On LSR-Synth–Chemistry, MOT-SR achieves 86.11% accuracy at tolerance 0.1 and an average NMSE of 3.85e-7, compared with LLM-SR (66.66%, 8.01e-06), LaSR (27.77%, 2.77e-04), and SGA (8.33%, 0.0458).

• Stronger generalization under held-out ID/OOD splits: On the test OOD split of Oscillator 1, MOT-SR reaches NMSE 6.20e-14, which the paper describes as nearly eleven orders of magnitude lower than LLM-SR's 1.4e-3. The authors attribute this to explicitly optimizing on a training-derived OOD region plus the meta-strategy module's variable-level and structural diagnostics. The ID/OOD test partitions are used only for final reporting and are not involved in parameter fitting or discovery-loop objectives.

• Faster convergence: MOT-SR reduces error more rapidly and converges to lower final NMSE than LLM-SR; in most cases it outperforms LLM-SR's best results at 2000 iterations within the first 1000 iterations.

• Better multi-objective quality: On Oscillator 1, MOT-SR shows faster Hypervolume (HV) growth and faster Inverted Generational Distance (IGD) reduction than LLM-SR, which plateaus earlier with higher IGD.

• Multi-objective optimization is essential for interpretability: Removing it (the w/o MultiObj variant) recovers only 1 of 9 symbolic terms, versus 4 of 6 for the full MOT-SR.

• Both meta-strategy submodules contribute: Removing the data analysis submodule (w/o Data) causes a notable performance drop; removing the structure submodule (w/o Struct) harms generalization and accuracy; removing the whole generator (w/o Strategy) yields prompt-only generation with the lowest accuracy and stability.

• EMRI trajectory-level correction: Using 58 configurations for discovery and 30 held-out configurations, MOT-SR achieves mean case-level NMSE of 1.17e-03 on held-out cases versus 1.20 for the neural residual baseline (NN) and 3.13e-02 for LLM-SR. Across all 88 valid cases, MOT-SR is 2.69e-03 versus NN 1.48 and LLM-SR 1.86e-01. On extra-long trajectories (>10,000), MOT-SR is 1.40e-03 versus NN 2.81 and LLM-SR 8.56e-02. The abstract summarizes this as approximately three orders of magnitude lower than the neural residual baseline and 26.8 times lower than LLM-SR.

Methodology in Plain English

MOT-SR operates as a closed loop. Starting from an observed dataset of inputs and outputs, it keeps a pool of candidate equations together with their scores.

First, a Meta Strategy Generator (an LLM) looks at the current best equations and picks which analytical tools to run — for example, correlation measures to find dominant variables, Fourier or wavelet analysis to find periodicity, or causality tests to detect directional influence. It converts the tool outputs into plain-language guidance. In parallel, it studies the equations themselves: which substructures recur, where the residual errors cluster, and which symbolic components have not yet been explored.

Second, an Equation Generator (a second LLM) writes a new batch of candidate equations guided by that strategy. To avoid everyone converging on the same shape, it parses each current equation into an abstract syntax tree, measures how structurally distinct each candidate is from the others using a subtree-overlap score, and samples parents with a softmax biased toward structural diversity. Those parents serve as in-context examples.

Third, each new equation is scored on three things at once: error on the middle (in-domain) part of the training data, error on the surrounding (out-of-domain) part of the training data, and the number of nodes in its abstract syntax tree. Equations that are not worse on every objective and better on at least one are kept as non-dominated; the rest are pruned. Over iterations this builds and refines a Pareto front, and the loop repeats.

For the EMRI case study, the learning target is not the trajectory itself but the discrepancy between two local evolution-rate models of the dimensionless semi-latus rectum, Δṗ(p, e, η) = ṗ_FSI(p, e, η) − ṗ_PN5(p, e, η). The discovered symbolic correction is added back to the PN5 rate and integrated over time, so success is measured by how well the integrated trajectory matches the FSI reference on cases never used for fitting.

Why This Matters

The paper argues that symbolic regression is central to scientific discovery because it produces compact, interpretable equations rather than black-box predictors, and that current LLM-based SR methods are held back by uninformed search spaces and single-objective scoring. MOT-SR shows that adding tool-grounded structural priors and explicit multi-objective pressure (especially generalization) yields large gains in accuracy, compactness, and search efficiency — and that the resulting equations transfer to unseen physical configurations.

Real-world applications suggested by the paper's domains:

• Gravitational-wave astronomy: correcting approximate inspiral models for extreme mass-ratio inspirals, where small local errors accumulate over many orbital cycles and where space-based detectors (millihertz band) require long-horizon accuracy.
• Materials science: modeling piecewise nonlinear deformation behavior such as the Stress-Strain task.
• Biology and biotechnology: modeling multivariate biological dynamics with nonlinear couplings, as in the E. coli growth task.
• Chemistry: recovering nested chemical-kinetics expressions, as in the 36 LSR-Synth–Chemistry tasks.

Industry relevance: the framework is model-agnostic across LLM backbones (tested with the open-source LLaMA-3.1-8B and commercial GPT-4o-mini), and the code is released at https://github.com/wswbx/MOT-SR. Interpretable equations are directly useful in scientific simulation, control, and modeling pipelines where auditability matters.

Future Directions

• LLM-synthesized tools: The paper explicitly states that MOT-SR opens the possibility for LLMs to synthesize new tools, expanding the scope of SR research.
• Beyond AST length as a complexity proxy: The authors acknowledge that Python AST length may disproportionately penalize composite function calls and adopt it only as a computationally inexpensive proxy; the associated trade-offs are deferred to an appendix.
• Broader application of the EMRI correction: The discovered symbolic correction is validated on Schwarzschild eccentric equatorial orbits against the FSI reference; extending it to more general orbital configurations is a natural open question, though the paper does not report such experiments.
• Broader use of the full LLM-SRBENCH suite: The authors limited chemistry evaluation to LSR-Synth–Chemistry, noting the other three datasets are constructed variants of the four benchmarks already used, and citing practical hardware constraints.

Target Audience

Researchers and practitioners in symbolic regression, automated scientific discovery, and LLM-based scientific reasoning will benefit most from the framework design and benchmark results. Scientists working on computational modeling — particularly in gravitational-wave astrophysics, materials science, and chemical kinetics — will find the EMRI case study directly relevant. Machine learning engineers interested in multi-objective search, Pareto-front maintenance, and tool-augmented LLM agents will also find the methodology applicable beyond equation discovery.

Authors’ abstract

Symbolic Regression (SR) aims to discover analytical equations from observational data and plays a central role in scientific modeling. While recent Large Language Model (LLM) based approaches show promise, they face two limitations. First, they lack data analysis mechanisms for uncovering variable dependencies, which reduces the efficiency of equation discovery. Second, most methods rely on single-objective evaluation focused solely on fitting error. This neglect of structural complexity and generalization often causes models to converge prematurely to local optima, limiting their ability to explore the broader equation space. We propose Multi-Objective Tool-augmented Symbolic Regression (MOT-SR), a unified framework that integrates external analytical tools to extract structural priors and guide equation generation, while jointly optimizing for accuracy, complexity, and generalization via a multi-objective evaluation module that maintains a dynamic Pareto front. MOT-SR employs two collaborative LLM modules: a Meta Strategy Generator, which selects tools and synthesizes structural optimization strategies based on Pareto-optimal equations, and an Equation Generator, which produces new candidate equations accordingly. The system operates in a closed-loop manner, continuously refining both strategies and equation structures. Across 40 standard tasks, MOT-SR outperforms existing SR methods in accuracy, generalization, and efficiency. We further validate MOT-SR on extreme mass-ratio inspiral (EMRI) orbital modeling, an important problem in space-based gravitational-wave astronomy where small local errors can accumulate substantially over long-term evolution. The discovered interpretable correction achieves the lowest trajectory-level integration error on held-out configurations. These results demonstrate the potential of MOT-SR to enable reliable modeling of long-horizon scientific dynamics.