Müntz-Szász Networks: Neural Architectures with Learnable Power-Law Bases

AI summary

Overview

Research area: Machine learning, specifically neural network architecture design and approximation theory, with applications to scientific computing and physics-informed neural networks (PINNs).

Technical level: Advanced. The paper combines classical approximation theory (the Müntz-Szász theorem), proofs of universal approximation and approximation rates, and empirical benchmarks against MLP baselines.

Scope: The paper introduces and analyzes a neural architecture whose edges compute sums of power functions with learnable fractional exponents, and evaluates it on singular-function regression and two PINN benchmarks.

What This Paper Is About

Standard neural networks use fixed activation functions (ReLU, tanh, sigmoid) that are inefficient at representing functions with singular or fractional power behavior, such as √x, crack-tip stresses scaling as r^(−1/2), or corner singularities of the form r^(π/ω). The paper proposes replacing fixed activations with learnable power-law bases, so that the network learns the exponents (not only the coefficients) of power functions, guided by the classical Müntz-Szász theorem. The goal is to show that this theory-guided architecture achieves much better accuracy with far fewer parameters on functions with power-law structure, while remaining competitive (but not dominant) on smooth functions.

Key Contributions

1. Architecture: Müntz-Szász Networks (MSN), where each edge computes a Müntz polynomial φ(x) = Σ_k a_k |x|^(μ_k) + Σ_k b_k sign(x)|x|^(λ_k) with learnable exponents {μ_k, λ_k}. The paper introduces a bounded exponent parameterization for numerical stability and a "Müntz divergence regularizer" (R_Müntz = ReLU(C − D(μ, λ)), where D is the sum of reciprocals of the exponents).

2. Theory: A universal approximation theorem for MSN (Theorem 3), plus an approximation rate (Theorem 4) showing that for f(x) = x^α an MSN edge achieves error O(δ²) when the nearest learned exponent is within δ of α, with exactly zero error if some μ_j = α.

3. Ablations and stabilization: Training techniques including two-time-scale optimization (exponents at 0.02η, coefficients at η), exponent warmup, and separate exponent gradient clipping, plus ablations removing the Müntz regularizer and replacing the parameterization with a cumsum variant.

4. Experiments and interpretability: Supervised regression on singular targets and PINN benchmarks (a singular ODE and a stiff boundary-layer BVP), with learned exponents reported as interpretable and matching known solution structure.

Main Findings

• Supervised regression on √x: MSN achieves RMSE 0.00224 ± 0.0005 with 425 parameters, versus MLP (big) at 0.01043 ± 0.0028 with 4,353 parameters and MLP (param-matched) at 0.01788 ± 0.0004 with 438 parameters. The paper reports this as 4.6× lower error than MLP (big) and 8× lower than the parameter-matched MLP.

• Supervised regression on the cusp |x − 0.5|^0.2: MSN achieves RMSE 0.00500 ± 0.0007 with 1,489 parameters, versus MLP (big) 0.02175 ± 0.0046 (4,545 parameters) and MLP (param-matched) 0.02741 ± 0.0022 (1,549 parameters). Reported improvements are 4.4× and 5.5× respectively.

• Smooth-function control: On the smooth polynomial x³ + 0.5x⁷, MLP (big) achieves the best result (0.00206 ± 0.0003 with 4,353 parameters) versus MSN (0.00355 ± 0.0021 with 425 parameters). The paper states this confirms MSN's advantage is specific to singular structure and does not uniformly dominate.

• Ablation on the regularizer: Removing the Müntz regularizer ("MSN (no Müntz)") gives RMSE 0.00399 ± 0.0038 on √x versus 0.00224 ± 0.0005 for full MSN, which the paper describes as a 1.8× degradation. On the cusp task, however, MSN (no Müntz) reports 0.00427 ± 0.0005, which is lower than the full MSN's 0.00500 ± 0.0007.

• Cumsum ablation: The cumsum parameterization reports 0.00498 ± 0.0039 on √x, 0.00630 ± 0.0039 on the cusp, and 0.00576 ± 0.0021 on the sparse polynomial, described as slightly less accurate but more stable.

• PINN singular ODE (u′ = 1/(2√x), u(0) = 0, exact solution √x): MSN reports RMSE 0.0529 ± 0.036 with 825 parameters, versus MLP (big) 0.1677 ± 0.047 (4,353 parameters) and MLP (param-matched) 0.2960 ± 0.176 (838 parameters). The paper describes this as 3.2× lower error than MLP (big) and 5.6× lower than the parameter-matched MLP, with 5× fewer parameters in the table caption and 5.3× fewer in the results text.

• PINN boundary-layer BVP: At ε = 0.05, MSN reports 0.222 ± 0.076 (825 parameters), MLP (big) 0.270 ± 0.080 (4,353 parameters), and MLP (param-matched) 0.334 ± 0.110 (838 parameters). At ε = 0.02, MSN reports 0.416 ± 0.049, MLP (big) 0.472 ± 0.005, and MLP (param-matched) 0.473 ± 0.005. The paper reports 18% lower error at ε = 0.05 and 12% at ε = 0.02.

• Failure at high stiffness: At ε = 0.01, all methods exhibit numerical instabilities (NaN values), which the paper attributes to a known limitation of vanilla PINNs.

• Interpretability: Learned exponent distributions concentrate near μ = 0.5 for the √x target and for the singular ODE, matching the true exponent, with no post-hoc interpretation required.

• Approximation-rate comparison: For f(x) = |x|^α with 0 < α < 1, the paper states that ReLU networks with P linear pieces achieve error O(P^(−2)) with matching lower bound Ω(P^(−2)), that smooth-activation networks require Ω(N^(1/α)) neurons for O(1/N) error, and that MSN achieves error ε with a single exponent K = 1 when |μ₁ − α| = O(√ε). The introduction and abstract alternatively phrase the MLP cost as O(ε^(−1/α)) neurons.

• Configuration details: Unless otherwise specified, MSN used K_e = K_o = 6 exponents, bounded mode with p_max = 2.0 to 4.0, and layer-wise exponent sharing; for PINNs, warmup of 500–1500 steps, exponent learning rates of 0.02η to 0.1η, and gradient clip δ = 0.03 to 0.1. The regularizer threshold was set to C = 2, and the paper notes robust performance for C ∈ [1, 5] in preliminary experiments. All experiments used 3 random seeds and report mean ± std.

Methodology in Plain English

Instead of giving every neuron a fixed activation like ReLU or tanh, the authors let each connection in the network evaluate a small sum of power functions whose exponents are themselves trained parameters. The classical Müntz-Szász theorem says that power functions with suitable real exponents can approximate any continuous function on an interval, so learning the exponents lets the network discover the natural power structure of the target (for example, finding μ = 0.5 for √x).

Because exponents must stay positive and bounded, the network does not optimize them directly. Raw parameters are passed through a sigmoid, sorted, and rescaled into a fixed interval (ε, p_max − ε), which guarantees positivity, ordering, differentiability almost everywhere, and a bounded gradient. A regularizer based on the sum of reciprocals of the exponents softly pushes the learned exponents toward diverse, slowly growing configurations consistent with the Müntz divergence condition. Training uses a slow learning rate for exponents, a warmup phase where exponents are frozen so coefficients can settle, and separate gradient clipping for exponent parameters. Experiments compare the resulting network against two MLP baselines (one larger, one parameter-matched) on three regression targets and two physics-informed problems, and the paper's theory section derives the L² projection error formula for approximating x^α with a finite set of power functions.

Why This Matters

Impact on research: The work bridges classical Müntz approximation theory and modern neural architecture design, offering a proof that learns the basis exponents of the approximation space rather than only the coefficients. It also provides an explicit error formula showing zero error when a learned exponent matches the target power, and argues that the architecture is complementary to, rather than a replacement for, existing PINN improvements such as Fourier features, adaptive weighting, and domain decomposition.

Real-world applications:

• Boundary-layer problems in fluid dynamics, where the paper cites O(ε^(1/2)) thickness scaling.
• Fracture mechanics, where crack-tip stresses scale as r^(−1/2).
• Corner singularities in elliptic PDEs, where solutions behave as r^(π/ω) for corner angle ω.
• Anomalous diffusion and fractional PDEs, listed among the intended problem classes.

Industry relevance: Scientific machine learning pipelines that rely on PINNs for simulation could benefit from architectures that represent singular solutions with far fewer parameters, provided the exp-log cost of evaluating non-integer powers is acceptable. The interpretability of the learned exponents may also be useful in settings where the solution form is partially unknown, since the network exposes the power structure rather than hiding it in weights.

Future Directions

• Higher dimensions: The paper states that current experiments are 1D and that extension via tensor products or radial bases is ongoing work.
• Fractional PDEs: Applying MSN to equations involving fractional derivatives is listed as future work.
• Stiff problems: Because all methods failed at ε = 0.01 in the boundary-layer benchmark, the paper suggests curriculum learning or combining MSN with other PINN improvements.
• Optimization robustness: The discrepancy between the regularizer's benefit on √x (1.8× improvement) and its neutral-to-negative effect on the cusp task suggests the design of exponent regularizers and parameterizations remains an open question, as does the trade-off between accuracy and stability observed for the cumsum variant.

Target Audience

Researchers in machine learning theory and scientific machine learning who work on approximation rates, activation function design, or physics-informed neural networks; numerical analysts familiar with Müntz polynomials and Müntz-Galerkin methods; and practitioners applying neural networks to PDEs with boundary layers, crack singularities, corner singularities, or fractional operators. Readers without a background in approximation theory will find the theorem statements and proofs dense, though the experimental tables and the interpretability results are accessible on their own.

Authors’ abstract

Standard neural network architectures employ fixed activation functions (ReLU, tanh, sigmoid) that are poorly suited for approximating functions with singular or fractional power behavior, a structure that arises ubiquitously in physics, including boundary layers, fracture mechanics, and corner singularities. We introduce Müntz-Szász Networks (MSN), a novel architecture that replaces fixed smooth activations with learnable fractional power bases grounded in classical approximation theory. Each MSN edge computes $φ(x) = \sum_k a_k |x|^{μ_k} + \sum_k b_k \mathrm{sign}(x)|x|^{λ_k}$, where the exponents $\{μ_k, λ_k\}$ are learned alongside the coefficients. We prove that MSN inherits universal approximation from the Müntz-Szász theorem and establish novel approximation rates: for functions of the form $|x|^α$, MSN achieves error $\mathcal{O}(|μ- α|^2)$ with a single learned exponent, whereas standard MLPs require $\mathcal{O}(ε^{-1/α})$ neurons for comparable accuracy. On supervised regression with singular target functions, MSN achieves 5-8x lower error than MLPs with 10x fewer parameters. Physics-informed neural networks (PINNs) represent a particularly demanding application for singular function approximation; on PINN benchmarks including a singular ODE and stiff boundary-layer problems, MSN achieves 3-6x improvement while learning interpretable exponents that match the known solution structure. Our results demonstrate that theory-guided architectural design can yield dramatic improvements for scientifically-motivated function classes.