SURFACEBENCH: A Geometry-Aware Benchmark for Symbolic Surface Discovery

AI summary

Overview

Research area: Machine learning for scientific discovery, specifically symbolic regression (equation discovery) and benchmarking of AI methods for recovering mathematical laws from data. The paper sits at the intersection of symbolic reasoning, computational geometry, and large language model (LLM) evaluation.

Technical level: Intermediate. Readers will benefit from familiarity with symbolic regression concepts, basic differential geometry (surface representations), and standard machine learning metrics, but the paper is written so that the core argument is accessible.

Scope in one sentence: The paper introduces a benchmark of 183 surface equations across 15 categories and three representation forms (explicit, implicit, parametric), together with a geometry-aware evaluation protocol, and uses it to show that no current symbolic regression method generalizes consistently across these tasks.

What This Paper Is About

Symbolic regression aims to recover concise mathematical expressions that explain observed data, but nearly all existing benchmarks evaluate simple scalar functions of the form y = f(x) and score results with string matching or normalized mean squared error (NMSE). The authors argue these benchmarks miss what real scientific equations do: they involve multiple coupled variables, hidden coordinate systems, and geometric structure, and they can be written in many algebraically different but geometrically identical ways (a sphere can be implicit, explicit, or parametric). The goal of this work is to build a benchmark where discovery quality is judged by the geometry a predicted equation produces, not by how closely its text matches the reference.

Key Contributions

1. SurfaceBench benchmark. A large-scale, geometry-aware benchmark for symbolic surface discovery containing 183 analytically constructed, science-inspired surface equations spanning 15 structurally defined categories and three representation paradigms: explicit, implicit, and parametric. Each task includes variable semantics and synthetically sampled 3D data, and the benchmark is designed to stress symbolic composition, structural ambiguity, and representational non-uniqueness while mitigating memorization.

2. Geometry-aware evaluation framework. The paper integrates symbolic equivalence checks with object-space metrics (Chamfer and Hausdorff distances) and regression-style error (NMSE), so that functional fidelity can be assessed beyond algebraic syntax. Under this protocol, the authors evaluate evolutionary, neural, and LLM-driven frameworks and report exact recovery rates of only 4% for LLM-based frameworks and 6% for traditional methods.

3. Error taxonomy and diagnostic ablations. The paper decomposes failure modes into symbolic (structural) failures versus geometric (shape-level) failures, and runs targeted ablations on noise sensitivity, out-of-domain generalization, and the injection of domain-prior prompts.

Main Findings

• No method wins across representations. Empirical evaluation across evolutionary, neural, and LLM-driven frameworks reveals no current method achieves consistent performance across representation types.

• Explicit surfaces favor symbolic accuracy; implicit surfaces favor geometry. Explicit surfaces yield the highest Symbolic Accuracy, whereas implicit surfaces achieve the lowest geometric distances. On explicit surfaces, models often recover the correct structural family but fail to produce geometrically tight parameterizations, leaving Chamfer and Hausdorff distances high. On implicit surfaces, distance-driven search brings equations closer to ground-truth geometry even when the algebraic form is not exact.

• Parametric equations are largely unsupported. Parametric equations remain the most underexplored representation paradigm. Among the benchmarked methods, only OpenEvolve and PySR reliably handle multiple coupled equations (multi-output regression) within a single optimization pipeline. LaSR, SGA, and LLM-SR lack algorithmic support for parametric equation discovery and were therefore excluded from the parametric ablations.

• LLM methods show strong priors but weak calibration. LLM-based approaches exhibit strong structural priors but limited robustness in parameter calibration and multi-equation reasoning. On the parametric task (Table 2), OpenEvolve with GPT4o-mini reaches SA 0.07, NMSE 0.84, Chamfer 1.22, Hausdorff 3.98; with Llama-3.1-8B, SA 0.02, NMSE 1.38, Chamfer 2.96, Hausdorff 4.87; with Qwen3-8B, SA 0.04, NMSE 1.25, Chamfer 2.35, Hausdorff 4.59. The non-LLM baseline PySR achieves SA 0.10, NMSE 0.61, Chamfer 2.52, Hausdorff 5.53.

• Best LLM configuration on explicit/implicit tasks. In Table 1, OpenEvolve with GPT4o-mini reaches the highest reported Symbolic Accuracy on explicit surfaces (0.50) with NMSE 0.98, Chamfer 2.69, Hausdorff 4.88, and the highest implicit Symbolic Accuracy among LLM methods (0.12) with NMSE 0.71, Chamfer 1.85, Hausdorff 4.96. Among non-LLM methods, PySR reports the lowest explicit Chamfer (0.13) and Hausdorff (0.41) with SA 0.25 and NMSE 0.18.

• Dataset construction was designed against memorization. The pipeline applies controlled symbolic perturbations inspired by LLM-SRBench, including functional nesting (e.g., sin(x) → sin(x² + y²)), additive and multiplicative term blending, coordinate reparameterization (affine, polar, spherical substitutions), and operator substitution (e.g., replacing sin(x) with tanh(x) or (1 − e^−x)), then applies novelty and stability checks against Feynman and SRBench.

• Noise sensitivity. In an experiment on 13 representative equations with 1%, 5%, and 10% Gaussian noise across GPT-4o-mini and LLaMA-3.1-8B backbones, traditional non-LLM SR methods show comparatively modest changes in geometric error, while LLM-based methods degrade substantially. Hausdorff often increases more sharply than Chamfer, indicating noise manifests as localized worst-case defects rather than uniform surface drift.

• Out-of-domain generalization. OOD samples are defined strictly as a range shift: models fitted on inputs in [-5,5] along each axis are tested on the non-overlapping exterior bands [-10,-5] ∪ [5,10]. A consistent generalization gap appears, with many discovered expressions behaving as accurate local interpolants in [-5,5] but failing outside the training support. Disproportionately larger OOD increases in Hausdorff relative to Chamfer indicate localized structural breakdowns rather than uniform global drift.

• Domain priors help only marginally. When correct domain priors (coordinate charts such as spherical or cylindrical, conservation or symmetry properties, scientific field knowledge) are injected into prompts, LLM-based methods show only marginal improvement and remain inferior to non-LLM methods. Table 3 reports, for example, OpenEvolve improving on explicit Chamfer from 2.93 to 1.16 (Δ +1.77) and explicit Hausdorff from 4.98 to 4.16 (Δ +0.82), and implicit Chamfer from 2.41 to 1.82 (Δ +0.59), with implicit Hausdorff 4.99 to 4.82 (Δ +0.17). LLM-SR improves on explicit Hausdorff from 27.43 to 20.77 (Δ +6.66) and implicit Hausdorff from 8.30 to 5.07 (Δ +3.23). The authors note that incorrect or mismatched priors often lead to severe degradation.

• Failure taxonomy: search versus fitting. A search failure occurs when the discovered equation includes terms from the incorrect functional family (for example, polynomials instead of trigonometric terms). An equation-fitting failure arises when the model identifies the relevant symbolic families but fails to assemble them in the correct structural order or infer accurate constants. LLM-based methods succeed primarily when they identify the correct functional categories early; after approximately 200 iterations, it is rare for these models to meaningfully recover toward the correct structural family once an initial hypothesis diverges.

Methodology in Plain English

The authors built their benchmark in eight stages: (1) pick scientifically motivated domains that naturally produce continuous 3D surfaces (optics, fluid dynamics, electromagnetics, materials science, robotics); (2) select representative problems and their analytic surface equations in explicit, implicit, or parametric form; (3) estimate the empirical distribution of symbolic operators in the collected corpus so that later augmentations reflect realistic usage; (4) apply controlled symbolic perturbations to break memorization of canonical forms; (5) run a novelty check, symbolically simplifying each equation and verifying analytic solvability, numerical stability, and novelty relative to Feynman and SRBench; (6) run a stability check under boundary conditions and asymptotic limits; (7) sample points uniformly across a bounded 3D domain, with adaptive sampling density in regions of high curvature; and (8) have domain experts visually inspect every surface.

For evaluation, they compare predictions to ground truth three ways. First, Symbolic Accuracy uses an LLM-based equivalence check with algebraic simplifications and parameter rescalings, following LLM-SRBench. Second, NMSE measures pointwise fit, for comparability with prior scalar benchmarks. Third, and most distinctive, predicted and reference surfaces are both sampled into dense point clouds, aligned under a similarity transform (removing translation, rotation, and scale), and compared with Chamfer Distance (mean geometric fidelity, capturing global deviations) and Hausdorff Distance (worst-case deviation, sensitive to sharp discontinuities, holes, or missing components).

They then evaluate a suite of baselines. The LLM-based methods are LLM-SR (program search with multi-island evolutionary search), LaSR (concept learning blended with PySR-based evolutionary optimization), SGA (bilevel optimization alternating LLM-generated symbolic structures with PyTorch parameter fitting), and OpenEvolve (an open-source implementation of the AlphaEvolve framework using LLMs as rule generators with external fitness evaluation), each run with GPT4o-mini, Llama-3.1-8B, and Qwen3-8B backbones. The non-LLM methods are TPSR, NeSymReS, E2E, DSR, uDSR, PySR, and gplearn. Ablations probe noise robustness, range extrapolation, and domain-prior prompting.

Why This Matters

Impact on research. The paper reframes equation discovery from scalar regression to geometry-aware multi-output reasoning, and shows that current benchmarks — including synthetic ones like LLM-SRBench and textbook-derived ones like AI Feynman and SRBench — cannot detect the difference between an algebraically correct answer and a geometrically correct one. By providing a diagnostic testbed with a decomposition of failure modes, it gives the field a way to measure progress on structure discovery, parameter calibration, and multi-output reasoning separately rather than collapsing them into a single score.

Real-world applications (domains the paper draws on for surface construction):

• Optics: recovering analytic descriptions of lens and mirror surfaces where the governing form is naturally implicit or parametric.
• Fluid dynamics: identifying surface equations for free-surface and flow-boundary phenomena where variables are coupled.
• Electromagnetics: discovering field-aligned or level-set descriptions that are concise only in transformed coordinate systems.
• Materials science and robotics: representing shape and surface constraints where the same geometry admits multiple algebraic formulations and where extrapolation beyond sampled regions matters.

Industry relevance. Automated equation discovery is used in engineering design, simulation, and system identification pipelines. The finding that LLM-based frameworks degrade sharply under noise and range shift, and that only a small subset handle coupled multi-equation systems, is directly relevant to anyone considering deploying these tools on real experimental data rather than clean canonical formulas. The paper's own conclusion — that post-structure geometric calibration is a missing pipeline stage — points to a concrete engineering gap.

Future Directions

1. Add a geometric calibration stage. The results expose a pipeline gap: after structure discovery, a targeted geometric calibration step is needed to refine parameters such as scale and shift so that structural recovery translates into measurable gains in Chamfer and Hausdorff.

2. Support coupled multi-equation discovery. Few frameworks are explicitly designed to jointly learn coupled equation systems, which the paper identifies as essential for modeling parametric surfaces. Extending frameworks such as LaSR, SGA, and LLM-SR to parametric representations is an open problem.

3. Improve self-refinement in LLM-based search. Because LLM-based methods rarely recover after an initial hypothesis diverges (beyond roughly 200 iterations), the authors suggest future LLM-based frameworks should incorporate stronger post-structure calibration and iterative optimization mechanisms to mitigate the dominant equation-fitting failures.

4. Make better use of structural priors. Since correct priors produced only marginal gains and mismatched priors caused severe degradation, it remains open how to translate structural cues into improved optimization outcomes reliably.

Target Audience

Researchers and practitioners in symbolic regression and machine learning for science who need a harder, less memorization-prone evaluation suite; developers of LLM-driven scientific discovery frameworks who want to know where their methods break (parameter calibration, multi-output coupling, noise and extrapolation robustness); and domain scientists in geometry processing, computer graphics, and engineering who care about recovering interpretable closed-form surface descriptions rather than black-box implicit models such as DeepSDF or Occupancy Networks. Graduate students entering the field will find the failure taxonomy and the explicit distinction between symbolic and geometric metrics particularly instructive.

Authors’ abstract

Equation discovery from data is a central challenge in machine learning for science, which requires the recovery of concise symbolic expressions that govern complex physical and geometric phenomena. Recent large language model (LLM) approaches have shown promise in symbolic regression, yet existing benchmarks predominantly evaluate low-dimensional scalar functions and rely on string-level or regression-based metrics that fail to capture structural and geometric equivalence. We introduce SURFACEBENCH, the first geometry-aware benchmark for symbolic discovery of three-dimensional surfaces. Unlike scalar curve-fitting tasks, SURFACEBENCH targets surface-level reasoning, where multi-variable coupling, coordinate transformations, and geometric structure must be inferred directly from data. The benchmark comprises 183 analytically constructed, science-inspired surface equations across 15 categories and three representation paradigms: explicit, implicit, and parametric forms. Each task includes variable semantics and synthetically sampled 3D data, and is designed to stress symbolic composition, structural ambiguity, and representational non-uniqueness while mitigating memorization. To evaluate discovery quality, SURFACEBENCH incorporates symbolic equivalence checks with geometric metrics of the object-space (Chamfer and Hausdorff distances) and regression-based error measures, allowing evaluation of functional fidelity beyond algebraic syntax. Empirical evaluation across evolutionary, neural, and LLM-driven frameworks reveals that no current method achieves consistent performance across representation types, with LLM-based approaches exhibiting strong structural priors but limited robustness in parameter calibration and multi-equation reasoning.The code and data are available at this link: github.com/deep-symbolic-mathematics/surfacebench.