Compute-optimal data scaling for neural surrogates via multi-fidelity training

AI summary

Overview

Research area: Scientific machine learning (SciML), specifically neural surrogates for partial differential equations (PDEs) and data scaling laws for surrogate training.

Technical level: Intermediate. The central idea is accessible, but the paper relies on numerical-analysis concepts (discretization error, solver error, convergence rates) and on the notation of surrogate architectures (FNO, U-Net, Transolver, AB-UPT).

Scope: A large empirical study (over 7,000 training runs and more than 12,000 GPU-hours) showing that mixing simulation fidelities under a fixed data-generation compute budget yields better surrogate accuracy per unit of generation cost than generating all data at a single high fidelity.

What This Paper Is About

Neural surrogates for PDEs are trained on synthetic data produced by numerical simulators, and as models and datasets grow, the cost of generating that data can outweigh the cost of training the model. Unlike images or text, each PDE sample has a tunable cost: coarser grids or looser solver tolerances produce cheaper but less accurate samples, with a cost-accuracy trade-off governed by known convergence rates. The paper asks whether, at the same total data-generation cost, training on a mixture of samples spanning many fidelity levels beats training on high-fidelity data alone.

Key Contributions

1. The authors formalize neural surrogate training as a function of both data quantity and fidelity under a fixed data-generation compute budget, exposing the trade-off between sample quality and dataset size, and treat fidelity as a fine-grained discrete spectrum rather than a binary low/high distinction.
2. They show empirically that neural surrogates learn the fidelity of their training data, and that conditioning on fidelity enables steering and interpolation across the fidelity spectrum at inference time.
3. They validate that multi-fidelity training at optimal mixtures produces Pareto-optimal data scaling, matching or outperforming single-fidelity training at matched data-generation compute across PDEs, discretization types, error sources, and surrogate architectures.
4. They report what they describe as the broadest empirical study of multi-fidelity neural surrogate data scaling to date, comprising over 7,000 training runs and more than 12,000 GPU-hours.

Main Findings

• Multi-fidelity training saves generation compute: Across the 20 setups summarized in the paper's cost-savings figure, multi-fidelity training yields cost savings in every case, up to 6.3× with a mean of ~3.1×, at matched test error. The best results appear in the low-to-medium budget regime (B̃ ≈ 100–500 equivalent high-fidelity samples).
• A spectrum beats a binary split: On the hyperelasticity dataset (discretization error) at B̃ = 200 with a Transolver model, treating fidelity as a spectrum (19 additional intermediate resolutions) achieves 33% lower error than a binary coarse/fine formulation, with a noticeably wider minimum in the mixture parameter λ. The gap narrows when the fidelity range is smaller, as for solver error on a fixed mesh.
• Fidelity conditioning enables inference-time steering: A surrogate trained at B̃ = 1,600 with λ* = −3.59 (Transolver, hyperelasticity, solver error) traces the simulator's error curve across the training support, interpolating at unseen interior fidelity values. An unconditioned baseline trained on the same data composition shows a single error level regardless of the fidelity input.
• The optimum is interior for finite budgets and converges upward: Claim C1 holds — for budgets below a crossover, the compute-optimal mixture is interior; claim C2 holds — as budget grows, the optimal mixture shifts toward higher fidelity and the multi-fidelity and high-fidelity curves converge.
• Solver fidelity is the more practical axis: In the hyperelasticity (Transolver) case at B̃ ≈ 400, multi-fidelity training reduces test error by 59% (solver) and 49.4% (discretization) while drawing 8× and 10.4× more samples than the high-fidelity baseline. Yet at a 1×10⁻¹ test error threshold, the discretization axis yields a 2.1× cost reduction while the solver axis yields 4.8×, despite the discretization cost ratio being far larger (138.7 vs. 9.5). The authors attribute this to the difficulty of generalizing across large resolution ranges and recommend the solver axis as a starting point.
• Poisson results: In 3D, the discretization menu spans a 216× cost ratio versus only 4× on the solver axis, giving 4.6× versus 2.7× multi-fidelity savings. At B̃ = 300, the 3D U-Net achieves 46.9% (solver) / 42.2% (discretization) error reduction while drawing roughly 3× / 6× more training samples than the high-fidelity baseline.
• Architecture sensitivity: Across 1D and 2D experiments, the U-Net reaches 2.0–3.1× lower terminal error than the FNO at the highest tested budget, and the FNO recovers only 57–90% of the U-Net's relative multi-fidelity gain.
• Airfoil (RANS) demonstration: Generating all samples at the highest fidelity consumed approximately 13,580 CPU core-hours. Multi-fidelity training reduces data-generation cost by 2.4× (Transolver) and 2.5× (AB-UPT) at a 2% average relative L2 error threshold, shrinking generation from roughly 2 days on a 64-core CPU to under a day; the paper elsewhere states this as turning ~2 days into ~19 hours. It also cuts 1,820 core-hours at matched 2% volumetric error. At B̃ ≈ 200, AB-UPT cuts relative L2 error by 54.1% on volume fields (drawing 2.9× more samples) and 60.3% on surface fields, and reduces relative lift-coefficient error by 54.9%.
• Linear elasticity peak savings: Up to 6.3× cost savings on the solver axis at a 10% threshold.
• Mixture extrapolation is feasible: Using the calibration Ansatz λ*(B̃) = a + b log(B̃), mean held-out regret is <5% on solver-error setups and ≤10% on mesh discretization settings. Per-dataset values (first, second, third held-out budget) are: linear elasticity 4.0/5.2/4.4 (solver) and 6.3/4.5/4.0 (discretization); hyperelasticity 2.3/3.4/3.9 (solver) and 9.0/10.0/4.3 (discretization); airfoil 0.0/1.7 (discretization; solver not studied for airfoils).

Methodology in Plain English

The authors start from a standard pipeline: a numerical simulator takes an input field at some spatial resolution and produces a solution, and each run carries a fidelity parameter controlling accuracy. Two error sources are studied — discretization error, where fidelity is resolution (grid points or mesh nodes) and error decays algebraically; and solver error, where fidelity is the number of iterations and error decays exponentially. Both reduce error at extra compute cost.

Rather than fixing a single fidelity, they define a set of admissible fidelities and sample from a softmax distribution over them: p(ξ̂ᵢ; λ, Ξ) = exp(λξ̂ᵢ) / Σⱼ exp(λξ̂ⱼ). Positive λ favors high fidelity; negative λ favors low fidelity; λ = 0 is uniform. Fidelity values are normalized and clamped so that λ is comparable across experiments with different units and magnitudes. A surrogate is then conditioned on the normalized fidelity in addition to the input field, and trained by empirical risk minimization on the dataset produced by that mixture. Evaluation uses the highest in-support fidelity; the conditioning is baked into the surrogate's weights so inference speed is unaffected.

The budget is expressed as a normalized budget B̃ = B/c(ξ̂max), and the compute-optimal mixture λ*(B) is the λ minimizing test error at each budget. Experiments span regular grids and unstructured meshes, four architectures (FNO and U-Net for grids; Transolver and AB-UPT for meshes), five equation settings listed in the paper's overview table (Poisson, linear elasticity, hyperelasticity, RANS airfoils) spanning 1D to 3D, and both error sources. Budgets are reported in algorithmic units (nodes or iterations) for the grid and mesh studies, and in CPU core-hours for the airfoil case. Seed means ± 1 standard deviation and bootstrap confidence intervals are reported. A post-hoc analysis fits λ*(B̃) = a + b log(B̃) from small budgets and tests whether it extrapolates to larger ones via a regret metric.

Why This Matters

Impact on research. Data generation, not training, is often the bottleneck for scaling neural surrogates. This paper reframes sample fidelity as a first-class design variable and shows that the standard practice of generating entire datasets at one fixed fidelity is suboptimal for any finite budget. It also supplies an unusually broad empirical comparison across PDEs, discretization methods (finite differences, finite volumes, finite elements), two error sources, and four architectures.

Real-world applications.

• External aerodynamics: shape optimization for drag and lift behavior around airfoils, where reduced generation cost shortens the amortization point at which a surrogate becomes cheaper than running simulations directly.
• Structural mechanics: linear and nonlinear (Neo-Hookean) elasticity problems on unstructured meshes, including derived quantities such as von Mises stress.
• Surrogate deployment in design loops, where cheaper dataset construction makes optimization pipelines cost-effective sooner.
• Foundation-scale scientific surrogates, whose compute budgets are increasingly dominated by simulation rather than training.

Industry relevance. The paper notes that established high-fidelity surrogate datasets such as DrivAerML, containing ~500 high-fidelity samples, sit squarely in the regime (B̃ ≈ 100–500) where multi-fidelity training delivers its largest gains. A ~3.1× average reduction in data-generation budget, and up to 6.3×, translates directly into lower simulation cost and faster dataset construction.

Future Directions

• Building a theoretical framework that combines the approximation behavior of neural surrogates (so far studied in restricted settings and at single fidelity) with the numerical convergence behavior of the underlying simulator, instead of relying on empirical calibration.
• Using active learning to discover optimal fidelity mixtures directly rather than extrapolating them from a handful of calibration runs.
• Extending the framework to transient dynamics — autoregressive surrogates at a fixed time step could reuse the existing fidelity axes, while the time step itself could become an additional fidelity axis if the architecture handles varying step sizes.
• Adding further fidelity axes from modeling error (for example turbulence closures and boundary layer treatment), and extending scalar fidelity conditioning toward a conditioning vector that captures the full simulation context for foundation models.
• Extending the approach to complex 3D geometries and industrial-scale meshes, and exploring more sophisticated sampling distributions (such as bimodal distributions), specialized fidelity fusion methods beyond conditioning, and training strategies such as fidelity scheduling or pretraining and finetuning.

Target Audience

Researchers and practitioners in scientific machine learning and computational science who train or deploy neural surrogates for PDEs, particularly those working on simulation-heavy problems such as computational fluid dynamics and structural mechanics. It is also relevant to groups budgeting large surrogate datasets, to methodologists interested in data scaling laws under controllable data quality, and to readers at an intermediate level who are comfortable with numerical discretization and solver terminology.

Authors’ abstract

Neural surrogates for Partial Differential Equations (PDEs) are usually trained on synthetic data from numerical simulators. As models grow and scaling laws demand larger datasets, the cost of data generation often outweighs the training cost and becomes the bottleneck in the scaling quest. Unlike data in other domains, however, each PDE sample has a tunable cost that depends on its numerical fidelity, with the resulting cost-accuracy trade-off governed by known convergence rates. We exploit this property by formalizing neural surrogate training as a function of both data quantity and fidelity under a fixed data generation compute budget, treating fidelity as a fine-grained discrete spectrum rather than a binary distinction. Building on this, we show that multi-fidelity training enables Pareto-optimal error scaling against data generation compute. By using simulations at lower fidelities as an additional training signal, training on the optimal fidelity mixture matches the accuracy of high-fidelity-only training, with up to $6.3\times$ less data generation compute across our settings. We validate these findings across six PDE problems spanning structured grids and unstructured meshes, two distinct error sources (discretization and iterative-solver error), and four surrogate architectures, totaling over 7,000 training runs.