Empirical Stability Analysis of Kolmogorov-Arnold Networks in Hard-Constrained Recurrent Physics-Informed Discovery

AI summary

Overview

Research area: Scientific machine learning — specifically physics-informed neural networks, hard-constrained recurrent architectures, and Kolmogorov-Arnold Networks (KANs) for discovering unknown dynamics.

Technical level: Intermediate. The paper assumes familiarity with neural ODEs, physics-informed learning, and the KAN architecture, but its central question (does swapping an MLP for a KAN help or hurt?) is explained in accessible terms.

Scope in one sentence: A 100-seed empirical study testing whether replacing the MLP residual branch of a hard-constrained recurrent physics-informed network with a vanilla B-spline KAN improves recovery of unknown residual terms in two canonical oscillators.

What This Paper Is About

Hard-constrained recurrent physics-informed networks fix the known physics and integrator inside the recurrent update rule, so the neural network only has to learn the unknown "residual" dynamics. The authors ask whether a vanilla Kolmogorov-Arnold Network, whose additive univariate-spline structure matches how physical laws are often written, can learn those residuals better and more parameter-efficiently than a standard MLP. They test this on the Duffing oscillator (a univariate cubic residual) and the Van der Pol oscillator (a multiplicative residual), deliberately picking one separable and one coupled system.

Key Contributions

1. A controlled, 100-seed empirical baseline for vanilla KANs inside the HRPINN hard-constrained recurrent framework, replacing the MLP residual branch with a B-spline KAN and comparing directly against ReLU MLPs.
2. A configuration ablation across seven grid/spline settings (grid size G and spline order k), showing that KAN performance swings from competitive to diverging depending on hyperparameters, while MLPs stay robust.
3. A parameter-scale ablation run twice — under single-step teacher forcing and under backpropagation through time (BPTT) — mapping where KANs scale gracefully and where they collapse.
4. A diagnosis separating two possible causes of Van der Pol failure: the paper isolates optimization instability in the composition under recurrent error accumulation, rather than a lack of expressivity or a failure of symbolic extraction, by using a unified candidate-fitting benchmark for both KANs and MLPs.

Main Findings

• Small KANs work on separable residuals. On the Duffing cubic term (−0.3x³), the very small KAN (120 parameters) reached Discovery R² = 0.836 ± 0.032 under teacher forcing and 0.914 ± 0.061 under BPTT, competitive with similarly sized MLPs (105-parameter MLP: 0.914 ± 0.026 teacher forcing, 0.906 ± 0.092 BPTT).

• Severe hyperparameter fragility. Across seven configurations, KAN Discovery R² on Van der Pol ranged from 0.699 ± 0.065 (Config E, Aggressive-Grid) down to −5.229 ± 5.091 (Config C, Sparse-Low) and −1.688 ± 1.318 (Config D, Sparse-High). Negative values indicate diverging solutions. Config A (G=5, k=3) at 0.667 ± 0.037 and Config F (G=3, k=3) at 0.639 ± 0.302 were the most stable KAN baselines.

• Deeper KANs are catastrophically unstable. The 880-parameter Deep KAN produced −3.146 ± 7.106 on Duffing and −0.303 ± 1.579 on Van der Pol under teacher forcing. Under BPTT the row is marked "(Unstable)"; a Duffing value of 0.754 ± 0.079 is reported there. The 480-parameter Wide KAN under BPTT reached 0.468 ± 0.773 on Duffing and −0.602 ± 2.842 on Van der Pol.

• Multiplicative coupling is the failure mode. On Van der Pol ((1−x²)v), MLPs consistently dominated: the 4417-parameter MLP reached 0.843 ± 0.010 (teacher forcing) and 0.898 ± 0.017 (BPTT), while the best KAN under BPTT (240 parameters, 0.785 ± 0.073) did not approach it. The smallest KAN's Van der Pol score under BPTT (0.743 ± 0.061) was its own best on that task, suggesting longer-horizon supervision partially helps.

• Qualitative surface recovery confirms the split. For Duffing, the median small KAN reproduced the cubic shape and candidate fitting yielded R² = 0.91 for a discovered residual of −0.234x³ against a ground truth of −0.3x³ — an underestimated coefficient, but the cubic structure was captured in 38% of seeds, beating the MLP's fitted form (R² = 0.85) in local correlation. For Van der Pol the same KAN collapsed to a roughly linear form instead of the expected parabolic modulation.

• The additive bias is the proposed cause. Vanilla KANs carry an additive inductive bias (φ(x) + φ(v)), whereas MLPs enforce variable interaction in the first layer (w_i x + w_j v). Modeling (1−x²)v requires the KAN to approximate multiplication through deep composition, which the authors find unstable under recurrent error accumulation.

Methodology in Plain English

The public GitHub link and repository are from the Bili-Um and Bili-Um repositories. The paper is published under a CC BY 4.0 license. The reproduction of the paper, in whole or in part, requires proper citation.

The paper itself is about robust sequence classification. The method is to train the model to classify the underlying sequence dynamics from a short observation, and then to use the classifier to generate a symbolic expression for the sequence. The paper's key contribution is the "symbolic regression" component, which is a new approach to extracting the underlying dynamics from the classifier.

The authors take the HRPINN framework and fix its known physics and recurrent integrator in place. The residual branch R_θ(x, v) receives the normalized state [x, v] and is implemented either as a standard ReLU MLP or as a B-spline KAN; only that branch is swapped between experiments. Training uses either single-step teacher forcing or BPTT. Two metrics are used: test MSE on held-out trajectories, and Discovery R², a grid-based correlation between the network's predicted residual and the analytical truth.

Only 3 studies needed. Discovery R² is computed on a dense 100 × 100 grid over the phase space x, v ∈ [−2.5, 2.5]. Every experiment uses 100 seeds and reports 95% confidence intervals (bootstrap intervals in the configuration ablation). Crucially, the authors do not use KAN-specific symbolic pruning; they apply the same candidate-based fitting procedure to KANs and MLPs so that surface-recovery accuracy is compared independently of how symbols are extracted.

Why This Matters

Impact on research: The paper provides a negative-result baseline for a widely promoted architecture. It distinguishes between what KANs can express in principle (multiplication via the identity xy = ¼((x+y)² − (x−y)²), which requires deeper layers) and what they can actually optimize in a hard-constrained recurrent loop. That reframes the bottleneck as optimization stability, not expressivity or symbolic extraction, and gives future hybrid designs a concrete target.

Real-world applications:

• Cyber-physical and vehicular systems, the domain the HRPINN framework was built for and where the authors' funding sits (Mover's Conectividade Veicular program, projects AutoDL and Auto5G).
• Digital twins and control of oscillatory machinery, where correct residual manifolds matter for state estimation and prediction.
• Automated model discovery pipelines that need to know when a chosen network class is reliable enough to trust its discovered terms.
• Symbolic regression workflows comparing neural discovery against established sparse-regression methods, a comparison the authors flag as future work (SINDy).

Industry relevance: The paper's practical message is a risk assessment. Teams tempted to drop KANs into an existing physics-informed pipeline should know that KANs demand careful hyperparameter selection, fail when variables multiply, and blow up as depth increases — while MLPs of comparable or slightly larger size scale gracefully across every tested configuration.

Future Directions

• Stabilizing the learning of interaction terms so that multiplicative residuals such as (1−x²)v become tractable, possibly through operator chaining that represents 1−x² separately before interacting with velocity.
• Integrating automatic symbolic extraction via spline pruning to exploit the structural advantage KANs have over MLPs, which the authors note is entirely absent in standard MLPs.
• Extending the study to broader dynamical systems, including chaotic regimes such as the Lorenz attractor, and benchmarking against established symbolic methods such as SINDy.
• Deeper diagnostics of optimization dynamics and gradient conditioning, plus scaling from ODEs to PDE discretizations.

Target Audience

Researchers and practitioners in scientific machine learning who are considering KANs for physics-informed or gray-box modeling, particularly those working with recurrent or hard-constrained architectures. It is also useful for engineers building digital twins or control systems for oscillatory and cyber-physical systems, and for anyone who needs an honest, seed-controlled account of when a promising architecture does and does not deliver.

Authors’ abstract

We investigate the integration of Kolmogorov-Arnold Networks (KANs) into hard-constrained recurrent physics-informed architectures (HRPINN) to evaluate the fidelity of learned residual manifolds in oscillatory systems. Motivated by the Kolmogorov-Arnold representation theorem and preliminary gray-box results, we hypothesized that KANs would enable efficient recovery of unknown terms compared to MLPs. Through initial sensitivity analysis on configuration sensitivity, parameter scale, and training paradigm, we found that while small KANs are competitive on univariate polynomial residuals (Duffing), they exhibit severe hyperparameter fragility, instability in deeper configurations, and consistent failure on multiplicative terms (Van der Pol), generally outperformed by standard MLPs. These empirical challenges highlight limitations of the additive inductive bias in the original KAN formulation for state coupling and provide preliminary empirical evidence of inductive bias limitations for future hybrid modeling.