EddyFormer: Accelerated Neural Simulations of Three-Dimensional Turbulence at Scale

AI summary

Overview

• Research area: Machine learning for scientific simulation, specifically neural surrogate models for computational fluid dynamics (turbulence).
• Technical level: Advanced. The paper assumes familiarity with the Navier-Stokes equations, large-eddy simulation, spectral element methods, and Transformer architectures.
• Scope: The paper introduces EddyFormer, a Transformer-based spectral-element architecture that simulates three-dimensional turbulence at DNS-level accuracy with a reported 30x speedup, and evaluates it on homogeneous isotropic turbulence, Kolmogorov flow, and The Well benchmark suite.

What This Paper Is About

Direct numerical simulation (DNS) of turbulence is extremely expensive because fully resolving a flow requires resolution scaling with Re^(9/4), making high-Reynolds-number simulation prohibitive even on supercomputers. The authors build a machine learning model that learns to correct the output of a cheap coarse numerical solver so that it matches high-resolution DNS, using a design inspired by the classical large-eddy simulation (LES) decomposition of flow into large-scale and subgrid-scale parts. The goal is accurate, stable, long-time turbulence prediction at a fraction of the cost of DNS.

Key Contributions

1. EddyFormer architecture: A Transformer-based spectral-element (SEM) model that splits prediction into an LES stream (global, coherent structures modeled with SEM-based attention) and an SGS stream (local eddy dynamics modeled with SEM-based convolutions). Each coarse element acts as an "SEM token," so the attention sequence length equals the number of elements N^3 rather than the far larger number of resolved modes N^3 M^3.
2. SEM tokenization: Inputs are decomposed into grid-scale (LES-filtered, spectrally truncated at k_max modes per axis) and subgrid-scale components, combining local spectral expressiveness with global attention scalability.
3. A new three-dimensional isotropic turbulence dataset (Re94): Generated with a high-accuracy pseudo-spectral DNS at 384^3 resolution (Kolmogorov scale of 0.5 eta), with the dataset recorded at 96^4 resolution.
4. Domain generalization mechanism: By tiling additional elements and applying a fixed 16 x 16 attention window at test time, the trained model generalizes to domains up to 4x larger than those seen in training while remaining consistent with the training setup.

Main Findings

• DNS-level accuracy at 256^3: On the Re94 three-dimensional homogeneous isotropic turbulence case (Reynolds number Re approximately 94 at Taylor microscale), EddyFormer achieves the accuracy of DNS at 256^3 resolution with a 30x speedup, providing essentially real-time simulation on a single NVIDIA A100 GPU.
• Direct prediction beats baselines: In the pure machine learning setting on Re94, EddyFormer with Legendre basis reaches 8.52% one-step relative error with 2.3M parameters, versus F-FNO at 22.5% (1.7M), FNO at 23.1% (17.6M), AViT at 26.6% (6.0M), TF-Net at 27.7% (6.3M), GNOT at 69.7% (6.2M), and ResNet at 87.2% (2.3M). The Chebyshev variant reaches 8.61%. All models were trained for 15k steps.
• Error reduction over baselines: EddyFormer reduces error by 30% compared to neural operator and Transformer baselines.
• Speed and cost trade-off: EddyFormer adds approximately 50% extra computational cost over the base DNS at 96^3 resolution. Simulation time is 4.86 seconds versus 152 seconds for DNS at 256^3; the coarse DNS at 96^3 takes 3.51 seconds.
• Accuracy versus reference DNS at 384^3: At t = 5, EddyFormer's L2 error is 18.2%, compared to 16.3% for DNS at 256^3 and 55.9% for DNS at 96^3.
• Physics-invariant metrics: EddyFormer matches DNS-level invariant statistics including energy spectra, correlation functions (evaluated for 5 <= t <= 10) and third-order structure functions over rollouts up to 20 seconds, where baseline F-FNO shows higher rollout error and a mismatched structure function.
• Vortex structure fidelity: Visualization of the Q-criterion at t = 5 shows EddyFormer captures both vortex cores of the reference DNS, while F-FNO fails to resolve the vortex at the center of the domain.
• Domain generalization: On the two-dimensional Kolmogorov flow KF4 (Re approximately 800 at Taylor microscale, forcing mode k = 4), EddyFormer matches DNS accuracy at resolutions between 512^2 and 1024^2 and generalizes to both 2x and 4x larger domains, matching the scaled energy spectra of high-resolution DNS, whereas F-FNO deviates in high-frequency modes on larger domains.
• The Well benchmark: EddyFormer achieves the best reported scores across the turbulence subsets tested. On MHD_64 it scores 0.1160 (Chebyshev) and 0.1240 (Legendre) versus 0.1633 for CNextU-net; on shear_flow 0.0837 (Legendre) versus 0.8080 for CNextU-net; on rayleigh_benard 0.1854 (Legendre) versus 0.6699 for CNextU-net; and on rayleigh_taylor_instab. it scores 0.1115 (Chebyshev) and 0.1243 (Legendre) while all baselines exceed >10 and fail to converge.
• Rayleigh-Taylor rollout: EddyFormer accurately predicts buoyancy-driven turbulent plumes up to a rollout of t = 30, while baseline FNO fails to converge to a physical solution.

Methodology in Plain English

The authors take the classical large-eddy simulation idea and turn it into a neural network design. Instead of having the network predict an entire flow field from scratch, they have it correct the output of a cheap, coarse numerical solver (a DNS at 96^3 resolution for the three-dimensional case, and 256^2 for Kolmogorov flow).

The flow field is represented using spectral elements: the domain is divided into coarse cubes (8^3 elements, with 13^3 modes per element for Re94), and each element carries a compact spectral expansion. Two parallel network streams process this representation. The SGS stream uses small localized spectral convolutions to capture fine-scale eddy behavior. The LES stream treats each coarse element as a token and applies self-attention across tokens to capture long-range interactions, with rotary position encoding applied to keys and queries based on global coordinates. The filtered LES field is truncated at k_max = 5 modes per axis as a spectral cutoff filter. The two streams are summed to produce the final prediction.

Because the number of tokens equals the number of elements rather than the number of resolved modes, attention cost stays manageable. The trained model is also trained with a 5-step rollout fine-tuning stage on Kolmogorov flow to control error accumulation over time. Training used 15k steps, and for the learned-correction experiments, four NVIDIA A100 GPUs for approximately one day of wall-clock time.

Why This Matters

• Research impact: The work shows that a carefully structured neural architecture can match DNS accuracy on three-dimensional turbulence, addressing the common criticism that machine learning surrogates have not consistently matched numerical solvers. It also offers a promising route to generalizing across domain sizes at test time without retraining.
• Real-world applications:
  • Climate and weather modeling, where turbulence parameterization is a persistent source of uncertainty.
  • Aerospace engineering, particularly aerodynamics and flow design.
  • Energy systems, including combustion and thermal-fluid design.
  • Environmental modeling, where simulation speed and accuracy affect resource-management decisions.
• Industry relevance: The 30x speedup and near-real-time simulation capability on a single GPU make long-time ensemble rollouts and interactive design workflows feasible where high-resolution DNS would be too expensive. The code and dataset are publicly released at https://github.com/ASK-Berkeley/EddyFormer.

Future Directions

• Extending to more complex flow domains: The authors state that future work will focus on extending EddyFormer to more complex geometries beyond the periodic domains tested.
• Scaling for extreme-scale turbulence: Further scaling of the architecture for higher-Reynolds-number flows is called out as an open direction.
• Validation depth: As noted in the broader impacts section, there is a risk of over-reliance on model accuracy without sufficient validation, and of biases in training data.
• Generalization of the attention-mask scheme: Although domain generalization to 4x larger domains is demonstrated on a two-dimensional case, the paper does not report whether this carries over to three-dimensional domains or to non-periodic boundary conditions.

Target Audience

This paper is most useful for machine learning researchers working on scientific computing and neural surrogates for partial differential equations, computational fluid dynamics practitioners interested in accelerating turbulence simulation, and researchers in climate, aerospace, or energy modeling who need fast but physically faithful flow predictions. Readers need background in fluid dynamics and modern sequence-model architectures to follow the architectural and experimental details; the paper also reports detailed baselines and benchmark tables that make it useful for those comparing methods on The Well suite.

Authors’ abstract

Computationally resolving turbulence remains a central challenge in fluid dynamics due to its multi-scale interactions. Fully resolving large-scale turbulence through direct numerical simulation (DNS) is computationally prohibitive, motivating data-driven machine learning alternatives. In this work, we propose EddyFormer, a Transformer-based spectral-element (SEM) architecture for large-scale turbulence simulation that combines the accuracy of spectral methods with the scalability of the attention mechanism. We introduce an SEM tokenization that decomposes the flow into grid-scale and subgrid-scale components, enabling capture of both local and global features. We create a new three-dimensional isotropic turbulence dataset and train EddyFormer to achieves DNS-level accuracy at 256^3 resolution, providing a 30x speedup over DNS. When applied to unseen domains up to 4x larger than in training, EddyFormer preserves accuracy on physics-invariant metrics-energy spectra, correlation functions, and structure functions-showing domain generalization. On The Well benchmark suite of diverse turbulent flows, EddyFormer resolves cases where prior ML models fail to converge, accurately reproducing complex dynamics across a wide range of physical conditions.