Neural Symbollic Regression Using Deep Learning and Sparse Modelling

AI summary

Authors: Ravi Kumar U, Sumitra S (Department of Mathematics, Indian Institute of Space Science and Technology, Thiruvananthapuram, India) arXiv: 2609.01102v1 [cs.LG], 01 Sep 2026 — License: CC BY-SA 4.0

Overview

• Research area: Symbolic regression / interpretable machine learning / scientific machine learning, combining deep learning, sparse regression, and distributed hyperparameter optimization.
• Technical level: Intermediate. The paper assumes familiarity with multilayer perceptrons, LASSO-style sparse regression, the SINDy formulation, and hyperparameter search schedulers (ASHA within Ray Tune).
• Scope: One-sentence scope: the paper proposes a decoupled neural–symbolic pipeline in which a neural network acts only as a noise-smoothing "functional preconditioner" over an interaction-aware nonlinear feature library, after which LASSO extracts closed-form expressions, and it evaluates this framework on the Nguyen-1 to Nguyen-7 benchmarks.

What This Paper Is About

Symbolic regression tries to recover short, human-readable mathematical formulas from data, but classical Genetic Programming is slow and noise-sensitive, while sparse-regression methods such as SINDy depend heavily on a hand-designed feature library and on how well that library captures nonlinear interactions. The authors ask whether a neural network can be used not to generate symbols directly (as in Deep Symbolic Regression or transformer-based approaches) but purely as a smooth, noise-robust approximation stage that prepares data for an independent sparse symbolic extraction step. The goal is to achieve better RMSE, noise robustness, and out-of-distribution generalization than SINDy and non-tuned neural baselines while retaining interpretable closed-form output.

Key Contributions

1. A decoupled neural–symbolic pipeline with explicit functional separation. Unlike hybrid methods such as AI Feynman and neural-guided symbolic regression that mix approximation with symbolic search, the network here functions solely as a smooth, noise-resistant functional approximator, followed by an independent sparse symbolic recovery phase. The authors state this division minimizes noise sensitivity and stabilizes symbolic extraction, particularly where nonlinear and transcendental elements are involved.

2. An interaction-aware feature library with controlled expressivity. The library combines base transformations with organized pairwise interaction terms rather than relying on rigid polynomial expansions, aiming to balance expressiveness against manageability while remaining compatible with sparse regression.

3. Neural preconditioning for sparse symbolic recovery. The paper argues that neural approximation can serve as a preconditioning phase that reduces observation noise and improves the clarity of symbolic structure, in contrast to methods that directly generate or optimize symbolic expressions.

4. Integrated hyperparameter optimization for neural–symbolic models. Distributed tuning with Ray Tune and ASHA scheduling is folded into the symbolic regression workflow; the authors state that systematic incorporation of tuning into neural-symbolic regression remains largely unexamined, and that tuning materially affects both predictive precision and symbolic integrity.

5. Empirical evaluation beyond standard benchmarks. Beyond the Nguyen suite, the study assesses robustness across noise levels, out-of-distribution generalization, and architectural/feature design choices through ablations.

Main Findings

• RMSE across Nguyen benchmarks (Table IV; results averaged over multiple runs): NSR (Tuned) achieves 0.0087 on Nguyen-1, 0.0099 on Nguyen-2, 0.0122 on Nguyen-3, and 0.0205 on Nguyen-4. SINDy obtains 0.0423, 0.0425, 0.0433, and 0.0463 respectively. PySR obtains 0.2111, 0.0802, 0.0798, and 0.0039.

• Where NSR wins and where PySR wins: NSR attains the lowest RMSE on Nguyen-1, Nguyen-2, and Nguyen-3; PySR excels on Nguyen-4, which the authors attribute to its evolutionary exploration of compact trigonometric forms, while noting PySR is less stable on noisy polynomial benchmarks.

• Symbolic recovery fidelity (Nguyen-3 example): The ground truth f(x) = x⁵ + x⁴ + x³ + x² + x is recovered as 1.002x⁵ + 0.998x⁴ + 1.001x³ + 1.000x² + 0.999x — the same structure with only minor numerical deviations.

• Training dynamics: Tuned NSR converges faster, reaches a lower minimum loss, and shows more consistent optimization than the baseline NSR, which the authors attribute to ASHA-driven hyperparameter selection.

• Noise robustness: Under Gaussian noise levels σ ∈ {0, 0.01, 0.05, 0.1}, tuned NSR consistently surpasses SINDy and degrades more gradually, which the paper links to neural smoothing prior to sparse extraction.

• Out-of-distribution generalization: Training on [-1, 1] and testing on [-2, 2] on Nguyen-4, NSR shows considerably better predictive consistency outside the training interval, whereas SINDy shows marked divergence.

• Ablation results (Table V): Full NSR (tuned) reaches RMSE 0.004 with 5 terms selected. Removing interaction terms gives 0.037 with 3 terms; a reduced feature library gives 0.142 with 2 terms; a shallow network (L = 1) gives 0.081 with 4 terms; and no hyperparameter tuning gives 0.019 with 7 terms.

• Runtime (Table III, averaged over five independent runs on identical hardware): SINDy 1.2 s, NSR Baseline 18.4 s, PySR 24.6 s, NSR Tuned 42.7 s.

• Complexity comparison (Table II): GP is listed at O(PGNS) time and O(PS) space; SINDy at O(Nk²) time and O(Nk) space; NeuralSR (Baseline) at O(EN(kH + H²)) time and O(Nk + H²) space; NeuralSR (Tuned) at O(TEN(kH + H²)) time and O(TH²) space. The overall NSR complexity is given as O(Nk + EN(kH + H²) + Nk²) time and O(Nk + NLH + kH) space.

• Qualitative model comparison (Table I): GP is rated High interpretability but Low scalability and Low noise robustness; SINDy High/Medium/Medium; DSR Medium/Medium/Medium; NSR (Ours) High/High/High.

• Not reported: Numerical values for the "symbolic accuracy" metric are not reported, nor are standard deviations or per-benchmark results for Nguyen-5, Nguyen-6, and Nguyen-7 in the tables. The baselines section names only SINDy and a non-tuned neural baseline, although PySR also appears in the results and runtime tables.

Methodology in Plain English

1. Build a nonlinear feature library. Eight base functions — identity, sin, cos, exp, log(1+x), x², x³, and tanh — are applied to each input dimension, and second-order pairwise products between different dimensions are added to capture cross-dimensional interactions. This yields a total of k = Fd + C(d,2)F² features for second-order interactions, constructed in parallel with Joblib. The full library has dimensionality O(kd + k²·C(d,2)).

2. Fit a neural network on that library. A multilayer perceptron takes the expanded feature matrix as input and predicts the target, minimizing mean squared error plus an L2 weight penalty. Depth is chosen from {1, 2, 3} layers and width from {32, 64, 128} units, with ReLU or GELU activations, dropout and weight decay, and the AdamW optimizer, trained on GPU via CUDA or the Apple MPS backend. The network's role is smoothing: turning noisy observations into a representation that is easier to sparsely recover.

3. Extract symbols with LASSO. After neural training, LASSO is applied once to the same feature library (β* = argmin ‖Φ_full(X)β − y‖² + λ‖β‖₁). Nonzero coefficients mark active symbolic terms; SymPy converts those active terms into an analytical expression. The authors motivate this with the classical compressed-sensing statement that β* can be recovered exactly if Φ_full satisfies the RIP condition.

4. Tune with Ray Tune and ASHA. Learning rate is sampled log-uniformly from 10⁻⁴ to 10⁻², depth from {1, 2, 3}, width from {32, 64, 128}, batch size from {32, 64, 128}, and LASSO λ from {10⁻⁴, 10⁻³, 10⁻²}. Twenty trials run in parallel, and the ASHA scheduler promotes or prunes trials based on whether a trial's validation RMSE falls below γ times the median.

5. Evaluate. Data uses the Nguyen-1 to Nguyen-7 benchmarks with n = 1000 samples drawn uniformly in [-1, 1]^d, optional Gaussian noise ε ~ N(0, 0.01²), and a 70% / 15% / 15% train/validation/test split. Comparisons are against SINDy (linear sparse regression over the same library) and a non-tuned neural baseline, with PySR also appearing in the results tables. Hardware included Apple Silicon MPS systems plus cloud NVIDIA T4 and A100 GPUs, using PyTorch 2.0, scikit-learn 1.3, Ray Tune 2.x, SymPy, Joblib, and Python 3.10.

Why This Matters

Impact on research. The paper reframes what a neural network contributes to symbolic regression: not symbol generation, but preconditioning for sparsity-based recovery. If that framing holds, it offers a middle path between pure sparse identification (fast, interpretable, library-dependent) and neural symbolic generation (flexible, but unstable and harder to interpret). The reported low RMSE, gradual noise degradation, and OOD extrapolation suggest interpretable equations can be recovered reliably without a combinatorial symbolic search — although the ablation table shows that removing any single component (interactions, library breadth, depth, tuning) degrades accuracy or yields less understandable expressions, so the gains appear to come from the whole pipeline rather than one trick.

Real-world applications (drawn from the paper's stated use cases):

• System identification of dynamical systems and control systems.
• Physics discovery — recovering governing equations from data (the "discovering physics" framing in the discussion).
• Engineering modeling and optimization of engineering processes.
• Analysis of real noisy scientific data such as climate data, biological signals, and engineering sensor data (listed as future application, not yet demonstrated).
• Identification of governing equations from trajectory data when integrated with physics engines or numerical solvers (also framed as future work).

Industry relevance. The paper positions NSR as practical for "extensive scientific tasks" by combining GPU acceleration, symbolic extraction, and automated hyperparameter tuning. The measured runtime, however, shows the trade-off concretely: NSR Tuned at 42.7 seconds versus SINDy at 1.2 seconds on identical hardware, so the model delivers improved robustness and generalization at a real computational cost. The stated scalability ceiling is the combinatorial growth of the feature library, which the authors identify as the framework's main memory bottleneck and a barrier for high-dimensional or large-scale datasets.

Future Directions

1. Adaptive and learned feature libraries. Replace the fixed function set with neural-guided basis selection, evolutionary library construction, or dictionary learning so the model builds and prunes functions by relevance, cutting library size and cost.

2. Transformer-based symbolic decoding. Move beyond LASSO over a fixed basis toward transformer-driven symbolic decoders that can produce syntactically correct equations extending past linear combinations of basis functions.

3. Scaling to high-dimensional systems. The exponential growth of interaction terms limits NSR in high dimensions; proposed remedies include sparse interaction identification, hierarchical or low-rank feature construction, and attention-based variable selection, targeting PDEs, biological networks, and control systems.

4. Optimization stability, simulators, and real data. The paper proposes curriculum-style progression for symbolic tasks, meta-learning to transfer hyperparameters across benchmarks, and loss functions designed for symbolic sparsity; it also proposes coupling NSR with physics engines or numerical solvers to identify governing equations from trajectories, and validating on real noisy datasets such as climate, biological, and sensor data.

Target Audience

Researchers and graduate students in interpretable machine learning, scientific machine learning, system identification, and computational physics who want symbolic regression with a deep-learning front end; practitioners who already use SINDy and want better noise tolerance and extrapolation while keeping closed-form output; and engineers applying equation discovery to control, dynamics, or sensor data. It is most useful to readers comfortable with regularized regression and neural network training, since the method's core insight — neural smoothing as a preconditioner for sparse recovery — is best appreciated with that background.

Authors’ abstract

Symbolic Regression (SR) seeks to find succinct mathematical expressions that represent the fundamental relationships within data, providing interpretability and scientific understanding that exceeds that of black-box models. Nevertheless, traditional methods like Genetic Programming face challenges with scalability and are highly sensitive to noise, while sparse regression techniques such as SINDy rely significantly on predetermined feature libraries. In this work, we present a Neural Symbolic Regression (NSR) framework that treats neural networks as functional preconditioners for symbolic discovery. Our approach uses a decoupled pipeline: a neural network first learns a smooth, noise-robust approximation of the target function in an interaction- aware nonlinear feature space. LASSO is then applied to extract sparse, interpretable closed-form expressions. To improve predictive accuracy and symbolic fidelity by integrating distributed hyperparameter optimization with Ray Tune and ASHA scheduling. Experiments on the Nguyen benchmark suite show that our approach consistently outperforms SINDy and non-tuned neural baselines in RMSE, noise robustness, and out-of-distribution generalization. Ablation studies confirm the significance of feature interactions, neural depth, and tuning strategies. In general, this study presents a scalable and understandable neural-symbolic framework, creating a solid link between neural approximation and the discovery of sparse equations for scientific machine learning.