PINNeval: A Comprehensive Evaluation Standard for Physics-Informed Neural Networks

AI summary

Overview

Research area: Scientific machine learning — specifically the training and benchmarking of Physics-Informed Neural Networks (PINNs) for solving partial differential equations (PDEs).

Technical level: Intermediate. The core ideas are accessible to anyone who has trained a neural network, though the paper includes formal PDE notation and assumes familiarity with optimizers, loss weighting, and sampling strategies.

Scope: The paper proposes PINNeval, an evaluation protocol for PINN methods built on a four-component decomposition of the training pipeline, and releases PINNpipe, a JAX package containing 18 methods and nine benchmark problems, which is used to run four case studies.

What This Paper Is About

PINN research has produced a rapidly growing catalogue of methods, but most benchmarks test one method at a time while leaving the rest of the training pipeline at a "vanilla" default that no longer reflects current best practice. The authors argue this makes published comparisons unreliable and hard to transfer, because components of a PINN pipeline interact and are often designed to be used together. Their goal is a standardized evaluation protocol that forces fair, competitive comparisons, plus an open-source implementation to make those comparisons reproducible.

Key Contributions

1. A four-component decomposition of the PINN training pipeline — domain-importance adaptation (where residuals are evaluated and how points are weighted), loss balancing (how multiple loss terms are weighted), neural network architecture (the hypothesis class), and optimization scheme (optimizers and schedules) — offered as an explicit basis for evaluation.

2. The PINNeval evaluation protocol, which treats the four components as a coupled system and requires that components not under investigation be instantiated competitively rather than minimally. It specifies per-component requirements (model specification, optimization scheme, loss balancing, domain-importance adaptation) and evaluation-level requirements (claim under evaluation, comparison conditions, metrics, numerical precision, robustness and statistical reporting, reproducibility and limitations).

3. The EvalCard documentation format, built on the card-based reporting style of Dhar et al. (2025), which records in a uniform layout the methods compared, the architectural and optimization choices held constant, and the evaluation-level conditions — so that claims can be traced and checked across studies.

4. PINNpipe, a JAX package whose modules mirror the four-component decomposition, containing 18 methods from the current literature and nine efficiently implemented benchmark problems, together with all reported experiments.

Main Findings

• Domain-importance adaptation is cheap but not uniformly helpful. Across Random-R, LAS, vRBA, R3, and RAD, the runtime overhead of all adaptive methods is small relative to Random-R, so cost does not constrain the choice. Random-R was best on KdV and RT and within a small factor of the best on Wave and Heat; LAS matched or outperformed Random-R on Wave and Heat, though the margin was small and inconsistent. The remaining three methods mostly performed worse than Random-R and LAS on those four PDEs, with RAD trailing on most of them. The authors note the adaptive methods were run at the hyperparameters proposed by their original authors, who had demonstrated gains in pipelines built on other optimizers, in particular Adam, whereas this study used SOAP.

• Quasi-Newton methods are practical for complex PDEs. Section 4.2 reports results that refute the claim of Wang et al. (2025) that quasi-Newton methods are impractical for complex PDE benchmarks. A two-stage MLP pipeline (SOAP followed by SSBroyden2) attained both a lower median relative L2 error and a markedly shorter runtime than the single-stage PirateNet-SOAP baseline.

• The gain comes from the optimizer stage, not precision or capacity. A single-axis ablation (Appendix G.4) found that float64 precision alone yields no measurable gain, and replacing PirateNet by the MLP under SOAP alone mostly increases the error. Adding the quasi-Newton stage to the same MLP at the same precision reduces the error by up to a factor of 1.7 × 10^4 (Heat). The authors conclude that the larger error of the MLP under SOAP alone is a trainability limit of single-stage SOAP optimization, not a capacity limit, and that float64 precision is a prerequisite of the quasi-Newton stage rather than an independent contributor. Feasibility depends on the MLP being moderate in size, since SSBroyden2 stores a dense inverse-Hessian approximation with O(d²) memory for d trainable parameters.

• Implementation efficiency on demanding PDEs. On Ginzburg–Landau, the PirateNet-SOAP pipeline in jaxpi required 746.0 minutes on the authors' hardware, whereas PINNpipe completed the same benchmark in 519.2 minutes despite using the more expensive LAS domain-importance adaptation in place of Random-R. Comparable speed-ups were observed on the Rayleigh–Taylor benchmark.

• No loss-balancing method dominates, but two clear trends emerge. Comparing GradNorm-based balancing (Wang et al., 2023), ReLoBRaLo, SoftAdapt (Heydari et al., 2019), GradNorm-MTL (Chen et al., 2018), and a fixed-weight baseline across nine benchmarks, GradNorm-based balancing performed best overall, achieving the lowest median error on five of nine benchmarks (Wave, Poisson, Burgers, GL, RT) and remaining competitive on the others. The fixed-weight baseline performed best on KS, AC, and KdV. Two methods showed robustness failures: SoftAdapt failed to train on Wave and KS, and GradNorm-MTL collapsed on KdV. Results are medians over 10 seeds with one standard deviation indicated.

• Methodological critique of prior benchmarks. PINNacle, the most closely comparable study, evaluates each variant as a single modification to the vanilla PINN baseline with other components at defaults, and conflates loss balancing with domain-importance adaptation into one comparison category. Other studies are criticized for missing relevant baselines (Bischof and Kraus (2025) omit the GradNorm-based method of Wang et al. (2023)), for non-competitive surrounding pipelines (Jeong et al., 2025), for unattributable joint effects and unreported precision (Toscano et al., 2026), and for excluding quasi-Newton methods outright (Wang et al., 2025).

Methodology in Plain English

The authors start by writing down the standard PINN objective and extending it so that individual collocation points can carry weights (domain-importance adaptation) and each equation's loss can carry a scalar weight (loss balancing). This gives a single formal umbrella that existing methods can be slotted into, and it separates the two components that prior benchmarks tended to blur together.

From there they define what "competitive" means: a component choice is competitive if the evidence does not show an established alternative that consistently achieves better accuracy at comparable cost, or comparable accuracy at lower cost — with evidence drawn from the literature or from controlled experiments on a comparable problem class using the same metrics. Checking a pipeline then reduces to four focused checks, one per component.

The protocol is operationalized as a checklist of ten reporting items plus an EvalCard summary sheet, so each experiment states its claim, its baselines, what was held fixed, and how the non-evaluated components were made competitive — along with relative L2 error, runtime, iteration count, hardware, numerical precision, number of seeds, variability, and any observed instabilities.

Experiments are run in PINNpipe. A single MLP with random Fourier features serves as the common anchor across all studies, so that differences are attributable to the component under test rather than to an uncontrolled change in network expressivity. Each case study then varies one component (or, in the optimizer-and-network study, a deliberate coupling of two) while holding the rest at competitive settings.

Why This Matters

Impact on research. The paper argues that a large share of published PINN comparisons are conditional on a weak surrounding pipeline, and demonstrates this concretely by overturning a published conclusion about quasi-Newton optimizers being impractical. Requiring competitive surroundings, coupled reporting, and cost measured in wall-clock time rather than iteration counts changes what counts as evidence in the field. The EvalCard format also addresses a specific failure mode the authors document: results that cannot be attributed to the proposed method because precision, loss balancing, and the proposed component were all changed at once.

Real-world applications (the paper frames PINNs as relevant across scientific computing and engineering; its benchmark suite covers chaotic and multi-scale problems, though it explicitly excludes complex geometries):

• Surrogate modeling for PDE-constrained simulation, where a mesh-free differentiable surrogate can serve forward, inverse, and parametric formulations of the same governing equations.
• Fluid-mixing and instability problems, represented in the suite by the Rayleigh–Taylor benchmark.
• Wave and transport phenomena, represented by the KdV and Wave benchmarks.
• Pattern formation and phase-separation-type dynamics, represented by the Ginzburg–Landau, Kuramoto–Sivashinsky, and Allen–Cahn benchmarks.

Industry relevance. Practitioners who adopt PINNs report slow convergence, hyperparameter sensitivity, and stagnation at unfaithful solutions for stiff, multi-scale, or high-dimensional problems. A protocol that forces competitive baselines and reports wall-clock cost gives engineering teams a realistic basis for deciding whether a PINN method is worth deploying, rather than a comparison against a baseline nobody would ship. The finding that architecture was not the limiting factor — but the surrounding training pipeline was — directly redirects engineering effort.

Future Directions

• Extending the protocol to inverse problems and operator learning. The authors restrict the present study to PINNs and forward problems, but argue that inverse formulations and neural-operator methods share enough pipeline structure that the protocol and EvalCard format apply with adjustments, and view this as a natural next step.

• Covering harder PDE classes and complex geometries. The benchmark suite covers real-world, chaotic, and multi-scale problems but does not include problems on complex geometries, and other challenging PDE classes remain outside its scope.

• Keeping the "competitive" reference current. The protocol depends on the surrounding pipeline being instantiated competitively; as the field evolves, the reference instantiation must be updated regularly. The authors state PINNeval is designed to make such updates straightforward, but that no set of rules can replace the community's ongoing effort to maintain a current notion of best practice.

• Re-examining hyperparameters under competitive pipelines. The domain-importance study ran adaptive methods at the values proposed by their original authors, who had validated them in pipelines built on other optimizers (in particular Adam) while this study used SOAP; retuning under competitive surrounding pipelines is an open question the results implicitly raise.

Target Audience

Researchers and practitioners who develop or benchmark PINN methods and need comparisons that transfer; reviewers and benchmark authors who want a checklist for what a fair PINN evaluation must report; and engineers applying PINNs to PDE surrogates who want guidance on which pipeline components — optimizer, precision, loss balancing, sampling — actually drive accuracy, and at what runtime cost. Readers interested in the general problem of benchmarking modular machine-learning systems, beyond PINNs specifically, will also find the protocol structure and EvalCard format reusable.

Note: the full numerical tables, per-benchmark standard deviations across seeds, iteration counts per stage, peak VRAM measurements, and the complete PDE definitions referenced in Appendices F through H are not reproduced in the content available here; the paper reports them in those appendices.

Authors’ abstract

The Physics-Informed Neural Network (PINN) literature has produced a rapidly expanding catalogue of methods. We divide a PINN training pipeline systematically into four components and observe that many existing benchmarks study these components in isolation. These components are domain-importance adaptation, loss balancing, neural network architecture, and optimization scheme. Moreover, the studies set the surrounding and not considered components of the pipeline to a…