Towards Scaling Laws for Symbolic Regression

AI summary

Overview

Research area: Machine learning — specifically scaling laws applied to transformer-based symbolic regression (SR), sitting at the intersection of neural scaling research, automated scientific discovery, and interpretable tabular modeling.

Technical level: Intermediate. Readers will benefit from familiarity with transformer encoder-decoder architectures, neural scaling laws from language modeling (Kaplan-style compute/parameter/data trade-offs), and the basics of symbolic regression. The paper is self-contained enough that an advanced beginner can follow the high-level findings.

Scope: A systematic empirical study of whether transformer-based symbolic regression obeys predictable power-law scaling with training compute, spanning five model sizes (6.5M–93M parameters) and three orders of magnitude in compute.

What This Paper Is About

Symbolic regression tries to recover the mathematical expression that generated a set of observed data — for example, discovering a formula rather than fitting a black-box predictor. Deep learning approaches based on pre-trained transformers have recently caught up with classical genetic programming, but essentially all prior work has tuned the training recipe while holding model scale fixed, and the authors state they are not aware of any symbolic regression model trained with more than roughly 100 million parameters. This paper asks whether SR behaves like language modeling: does performance improve predictably as a power law of compute, and are there consistent compute-optimal choices of batch size, learning rate, and data-to-model ratio?

Key Contributions

1. Power-law scaling demonstrated. The authors show that both solved rate and loss follow power-law trends with training compute across three orders of magnitude, using a controlled end-to-end transformer pipeline.
2. Compute-optimal hyperparameter trends. They identify systematic scaling behavior for optimal learning rate, batch size, and token-to-parameter ratio, including an optimal token-to-parameter ratio of approximately 15 in their compute regime.
3. A scalable end-to-end pipeline. They introduce a two-step synthetic data generation scheme with canonicalization and deduplication, plus a per-cell embedding architecture with row- and column-wise attention, enabling clean scaling analysis.
4. Practical heuristics for future models. The findings are framed as design principles for the next generation of SR models rather than as a new state-of-the-art baseline — the authors explicitly state their goal is not to beat existing methods.

Main Findings

• Solved rate scales as a power law with compute. Using a Pareto front built by binning compute into 1,500 intervals and keeping the lowest-validation-loss models up to their compute level, Acc_solved rises from approximately 0.03 at the lowest compute budget to 0.6 at the highest. Extrapolating the fitted scaling law, the authors project reaching 0.8 at a compute budget of 3.8 × 10²¹ FLOPs.

• Loss and the relaxed accuracy metric scale similarly. Validation loss and Acc_{R²>0.99} also follow power-law trends with compute. Improvements in Acc_{R²>0.99} occur much faster than in Acc_solved, which the authors highlight as evidence of how hard exact expression matching is.

• No saturation observed. Larger models with more data consistently improved the solved ratio and reduced loss, with no signs of saturation in the largest models tested.

• Optimal batch size and learning rate grow with compute. Following a two-step Akima interpolation strategy, both compute-optimal batch size and learning rate increase as compute grows. The upward learning-rate trend contrasts with findings for large language models.

• Optimal token-to-parameter ratio of approximately 15. Fitting scaling laws to optimal parameter counts and token counts per compute budget, the ratio of the two predicted laws shows an increasing trend with compute — meaning training data size should scale slightly faster than model size, even though ≈15 is optimal within the studied compute range.

• Concrete per-model results. In the full results table, the smallest model (6.5M parameters, batch size 32, learning rate 4.6e-4) goes from Acc_solved of 0.0327 at 4.50e+15 FLOPs to 0.1490 at 7.20e+16 FLOPs; the largest (93M parameters, batch size 256, learning rate 1.0e-3) reaches Acc_solved of 0.5967 and validation loss of 0.1047 at 1.47e+19 FLOPs.

Methodology in Plain English

Data generation. Rather than sampling expressions randomly, the authors build expressions recursively: starting from variables alone, they repeatedly apply a set of binary operators (+, -, ·, ÷) and unary operators (exp, sin, neg, sqrt) to produce a complete set of expressions up to a fixed tree depth (depth 3, with two variables). Every expression is canonicalized and deduplicated so the model does not see the same function twice or in inconsistent forms. From a base set of 100,000 expressions, they sample up to 3,600 expression-dataset pairs each by inserting random integer constants (range -9 to 9, with probability 0.2) and generating a 64-point dataset from a Gaussian mixture, with up to five retries per attempt. Independent validation and test splits of 1,000 expressions are generated with fresh constants and datasets so no training pair appears in evaluation.

Model and training. Inputs are encoded in base-10 floating-point notation, splitting each value into a mantissa (including sign) and an exponent; target expressions are written as LaTeX strings with constants tokenized digit by digit. The architecture is a standard encoder-decoder transformer, but instead of merging each data point into one embedding, the model creates one embedding per table cell and applies both row-wise and column-wise attention in each layer — an idea borrowed from recent tabular foundation models. The decoder cross-attends to the target-cell embeddings only. Training uses cross-entropy loss against the true expression tokens, the AdamCPR optimizer, a linear learning-rate warm-up over the first 5% of steps, and cosine annealing thereafter.

Scaling protocol. Training compute is approximated by adapting the Kaplan et al. FLOP estimate to encoder-decoder models: FLOPs ≈ 6 · (N_enc · D_in + N_dec · D_out), where N is the number of feed-forward parameters and D_in, D_out are input and output token counts. Each model size is swept over batch sizes and learning rates at a token-to-parameter ratio of 20 (a value found optimal for language models), then retrained at the best configuration with token-to-parameter ratios from 5 to 80. Evaluation samples 128 candidate expressions from the model, keeps the one with the highest R² score, and reports Acc_solved and Acc_{R²>0.99} on the test split, averaged over three seeds.

Why This Matters

Impact on research. This is, according to the authors, the first systematic scaling study in symbolic regression. It reframes SR progress as a compute-allocation problem rather than a purely architectural or loss-engineering one, and it provides the first empirical evidence that transformer SR follows predictable power laws. It also surfaces a task-specific divergence from language modeling — the compute-optimal learning rate rises with model size here — suggesting that scaling recipes cannot simply be copied across domains.

Real-world applications.

• Scientific discovery: recovering governing equations from experimental measurements in physics, chemistry, and biology, where an interpretable formula is more valuable than a black-box fit.
• Interpretable tabular modeling: producing inherently understandable and generalizable models for tabular data, where exact expressions aid auditing and trust.
• Engineering and materials design: deriving compact empirical relationships from simulation or sensor data where practitioners need closed-form surrogates.
• Compute budgeting for ML teams: using the reported trends to decide how to allocate a fixed compute budget across model size, dataset size, batch size, and learning rate for SR training runs.

Industry relevance. Organizations training foundation models for tabular data can use the token-to-parameter ratio and hyperparameter scaling trends as practical heuristics. Because SR models are small by modern standards (the largest here is 93M parameters, and prior work stays under roughly 100M), these findings matter for teams that need interpretable models but cannot afford frontier-scale compute.

Future Directions

• Extend beyond the restricted expression space. The current setup limits expressions to at most two variables and small integer constants; real-world SR tasks typically involve more variables and floating-point constants, so whether the fitted laws transfer is untested.
• Verify extrapolations beyond the studied compute range. The authors note their analysis spans a limited compute range, so the projected 0.8 solved rate at 3.8 × 10²¹ FLOPs remains unverified.
• Reduce measurement variance. The authors state that more training runs over multiple seeds are needed to reduce variance in the hyperparameter interpolation trends, and that their training runs are single-seed due to computational constraints.
• Test whether scaled end-to-end SR can beat all other approaches. The paper does not compare against other SR methods, so a natural follow-up is to check whether improved data generation, architecture, and scaling together can outperform genetic programming and other baselines under a fair evaluation protocol.

Target Audience

Researchers and practitioners working on symbolic regression, neural scaling laws, and tabular foundation models. It is especially useful for graduate students and applied scientists who want to understand where compute should be spent when training transformer-based SR systems, and for method developers deciding whether scaling is a viable path to better interpretable models. Readers primarily interested in applied equation discovery on high-dimensional, floating-point, many-variable problems should treat the reported scaling laws as a starting hypothesis rather than a ready-to-transfer recipe.

Authors’ abstract

Symbolic regression (SR) aims to discover the underlying mathematical expressions that explain observed data. This holds promise for both gaining scientific insight and for producing inherently interpretable and generalizable models for tabular data. In this work we focus on the basics of SR. Deep learning-based SR has recently become competitive with genetic programming approaches, but the role of scale has remained largely unexplored. Inspired by scaling laws in language modeling, we present the first systematic investigation of scaling in SR, using a scalable end-to-end transformer pipeline and carefully generated training data. Across five different model sizes and spanning three orders of magnitude in compute, we find that both validation loss and solved rate follow clear power-law trends with compute. We further identify compute-optimal hyperparameter scaling: optimal batch size and learning rate grow with model size, and a token-to-parameter ratio of $\approx$15 is optimal in our regime, with a slight upward trend as compute increases. These results demonstrate that SR performance is largely predictable from compute and offer important insights for training the next generation of SR models.