Efficient Dilated Squeeze and Excitation Neural Operator for Differential Equations

AI summary

Overview

Research area: Machine learning for scientific computing — specifically neural operator learning for solving partial differential equations (PDEs), with an emphasis on inference and training efficiency.

Technical level: Intermediate. Readers should be comfortable with convolutional networks, attention mechanisms, and the general framing of PDE solving as learning a function-to-function mapping, but the paper's core ideas are explained through standard building blocks.

Scope: The paper introduces and empirically evaluates D-SENO (Dilated Squeeze-Excitation Neural Operator), a fully convolutional surrogate model benchmarked against more than ten neural operator and transformer-based PDE solvers across four PDE datasets.

What This Paper Is About

Numerical PDE solvers (finite difference, finite volume, spectral, finite element) are accurate but their cost explodes with mesh resolution, geometric complexity, and parametric sweeps, so the field has turned to learned surrogates. Existing high-accuracy surrogates such as transformer-based models and Fourier-based neural operators are parameter-heavy, making training costly and deployment sluggish, while convolution-based operators use fixed receptive fields that do not adapt to the physics. The goal of this paper is a lightweight, strictly local, fully convolutional operator that captures long-range physical dependencies and channel-wise feature relevance without Fourier transforms, self-attention, or positional encoding.

Key Contributions

1. A unified architecture combining dilated convolutions with squeeze-and-excitation. D-SENO pairs residual dilated convolution blocks with SE channel attention, capturing wide receptive fields and multiscale spatial structure alongside input-adaptive channel recalibration, all inside a fully convolutional design that preserves spatial resolution and scales linearly with input size.

2. Dataset-specific, direction-dependent dilation schedules. Unlike the Dilated Convolutional Neural Operator (DCNO), which applies all dilation rates in every processing block, D-SENO selects a single dataset-specific dilation rate per block and connects blocks via residual links. Dilations are applied independently along the x- and y-axes, allowing non-uniform, orientation-specific receptive field growth.

3. A refined FNO baseline. The authors report that modest tweaks to FNO — replacing ReLU with GELU, removing batch normalization, preserving sufficient Fourier modes, and implementing complex Fourier multiplication directly with einsum — yield notable gains in accuracy, speed, and parameter counts. This variant, FNO⁺, outperforms many neural operator variants introduced after the original FNO.

4. Benchmarking on four PDE problems with ablations. D-SENO is evaluated on airfoil potential flow, Poiseuille pipe flow, heterogeneous Darcy flow, and time-dependent incompressible Navier–Stokes, with an ablation study over depth, SE blocks, dilation configuration, projection width, and (for Darcy only) dataset resolution.

Main Findings

• Training speed. D-SENO achieves training speed up to approximately 20 times faster than standard transformer-based models and neural operators. In the cross-model table, D-SENO trains at 1.99 s per epoch on Airfoil, 3.30 s on Pipe, 0.93 s on Darcy, and 7.04 s on Navier–Stokes, versus Transolver at 28.03 s, 41.00 s, 18.47 s, and 245.10 s respectively. The conclusion states speed-ups of "multiple orders of magnitude," a different formulation than the approximately 20 times reported in the abstract.

• Accuracy against the prior state of the art. D-SENO outperforms Transolver on three of four benchmarks: relative L2 error 0.0052 vs. 0.0053 on Airfoil, 0.0030 vs. 0.0033 on Pipe, and 0.0048 vs. 0.0057 on Darcy.

• The Navier–Stokes exception. On the time-dependent Navier–Stokes torus dataset, FNO⁺ achieves lower relative L2 error than D-SENO (0.1054 vs. 0.1391). The authors attribute this to the periodic boundary conditions of the unit torus aligning naturally with Fourier-based representations, and to D-SENO not explicitly exploiting temporal periodicity.

• Broader baseline comparison. D-SENO's 0.0052 on Airfoil is lower than every listed baseline in the table, including LSM (0.0059), ONO (0.0061), HT-Net (0.0065), and FactFormer (0.0071). Its 0.0030 on Pipe is lower than GNOT (0.0047), LSM (0.0050), ONO (0.0052), and U-FNO (0.0056). Its 0.0048 on Darcy is lower than LSM (0.0065), FNO and geo-FNO (0.0108), and U-NO (0.0113).

• FNO⁺ validates the baseline refinement. FNO⁺ reaches 0.0057 (Airfoil), 0.0072 (Pipe), 0.0070 (Darcy), and 0.1054 (Navier–Stokes), improving on the original FNO's reported 0.0108 on Darcy and 0.1556 on Navier–Stokes (FNO's Airfoil and Pipe values are not reported in the table).

• Ablation on SE modules. Removing the SE modules leads to only a slight drop in performance in the main text, though the Airfoil ablation shows Model Airfoil-G w/o SE at 0.0056 versus 0.0052 with SE, alongside a reduction in parameters from 1.042 M to 0.984 M and time per epoch from 1.99 s to 1.70 s. On the Pipe dataset, Model Pipe-G w/o SE matches Model Pipe-G at 0.0030 with fewer parameters (1.176 M vs. 1.306 M) and lower time per epoch (3.30 s vs. 4.28 s).

• Depth sensitivity. Increasing the number of DS blocks lowers relative L2 error. On Airfoil, error falls from 0.0131 with 1 block to 0.0052 with 7 blocks; on Pipe, from 0.0107 with 1 block to 0.0030 with 7 blocks. Both gains come with increased parameters and per-epoch training time.

• Qualitative behavior. D-SENO resolves steep Mach transitions near the airfoil surface more sharply than FNO⁺, confines its largest Pipe velocity errors to the downstream end at lower magnitudes, and shows minimal underestimation on Darcy flow where FNO⁺ systematically over-predicts pressure in high-permeability regions.

• Ablation parameters studied. The study covers (i) depth, (ii) the squeeze-and-excitation block, (iii) dilation configuration, (iv) projection width, and (v) dataset resolution (Darcy only), applied to D-SENO, FNO, FNO⁺, and Transolver.

Methodology in Plain English

The researchers frame PDE solving as learning a solution operator: a network takes an input function (such as a geometry or a coefficient field) and outputs the corresponding solution field, generalizing to inputs unseen during training.

D-SENO follows a lift–process–project structure. A pointwise convolutional layer lifts the input into a high-dimensional latent space. A sequence of Dilated-Squeeze (DS) blocks processes it. A final pointwise layer projects to the output solution.

Each DS block starts with a dilated convolution block of two stacked convolutions (typically 3×3) using dilation rates chosen per dataset and applied separately along the x- and y-axes. Dilation inserts spacing between kernel elements, so the filter covers a larger area without adding parameters or reducing resolution; direction-dependent rates let the receptive field stretch to capture elongated, orientation-specific structures. After the convolutions, a modified SE block performs global average pooling to produce one summary statistic per channel, passes it through two 1×1 convolutions separated by GELU (with a reduction factor r controlling the intermediate width), and uses a sigmoid to produce channel weights that rescale the feature maps. A residual connection adds the block input to the recalibrated output, followed by a GELU activation.

No striding or downsampling is used, and global average pooling appears only inside SE blocks to compute channel weights without altering spatial dimensions. The authors note that dilation schedules are chosen per dataset rather than learned automatically.

Evaluation setup. All D-SENO experiments ran on a single NVIDIA A100 80 GB PCIe GPU, with performance numbers averaged over three independent random seeds. The primary metric is relative L2 error on held-out test sets. Training used 500 epochs, batch size 20, learning rate 1×10⁻³, step 100, decay 0.5, and weight decay 1×10⁻⁴ for every benchmark; latent widths were 64 (Airfoil), 96 (Pipe), 48 (Darcy), and 64 (Navier–Stokes), with the same hyperparameters used for D-SENO, FNO⁺, and FNO.

Datasets. Airfoil uses 1,000 training and 200 test samples on a 221×51 mesh, with 221×51×2 geometry inputs and 221×51×1 Mach number targets, generated by deforming a baseline NACA-0012 profile. Pipe uses 1,000 training and 200 test samples on a 129×129 mesh, with 129×129×2 geometry inputs and 129×129×1 horizontal velocity targets, generated by varying a randomly curved centerline. Darcy uses 1,000 training and 200 test samples, originally on a 421×421 grid and downsampled to 85×85, with 85×85×1 binary medium structure inputs and 85×85×1 pressure targets. Navier–Stokes models incompressible viscous flow on a unit torus with viscosity 10⁻⁵, using 1,000 training simulations and 200 test trajectories on a 64×64 grid, where a 64×64×2×10 tensor of two velocity components over the past 10 time steps predicts the same shape for the next 10 steps.

Why This Matters

Impact on research. The paper argues that strict locality — no Fourier transforms, self-attention, or positional encoding — is sufficient to capture domain-scale interactions in fluid and porous-media flows while reducing training time complexity. It offers a simple, computationally efficient baseline and building block for future neural solver development, and its FNO⁺ finding suggests that some reported gains of newer architectures may partly reflect baseline tuning choices.

Real-world applications. The paper motivates the work with fields where fast, accurate PDE surrogates are essential, including:

• Aerodynamics, including flow around deformed NACA-0012 airfoils.
• Porous media design and heterogeneous Darcy flow in subsurface systems.
• Flow control and Poiseuille pipe flow with varying centerline geometry.
• Vortical field prediction, including incompressible Navier–Stokes dynamics on a unit torus.

Industry relevance. Because D-SENO relies exclusively on standard convolution operations, it can execute on existing, highly optimized hardware without specialized machinery for spectral or attention layers. The reported per-epoch training times in the range of roughly 1–7 seconds, against 18–245 seconds for Transolver in the same comparison table, point to practical deployment where repeated retraining or rapid surrogate iteration is needed. The authors note that a single change in geometry or forcing profile can require substantial fine-tuning for physics-informed approaches, which makes low-cost operator retraining attractive.

Future Directions

• Address the Navier–Stokes gap. The authors suggest integrating periodic temporal kernels or expanding D-SENO's spectral receptive field, since periodic boundary conditions on the unit torus favor Fourier-based representations and D-SENO does not explicitly exploit temporal periodicity.

• Extend to unstructured meshes. The current evaluation focuses on structured-grid datasets; extending the framework to unstructured meshes and more complex geometries via masked or graph-based dilation is listed as an important next step.

• Learn the dilation schedules. Dilation rates are currently chosen per dataset by hand. Automatically learning or adapting dilation patterns could improve robustness, with potential trade-offs in compute and accuracy.

• Add theory and global context. The study is primarily empirical, and a complementary theoretical analysis of approximation, stability, and generalization is called out as needed. The authors also propose integrating physics-informed priors or constraints and combining D-SENO with attention modules to better capture long-range dependencies.

Target Audience

This paper is most useful for machine learning researchers working on neural operators and scientific machine learning surrogates, especially those interested in efficiency-accuracy trade-offs and in convolution-only alternatives to spectral and attention-based operators. Practitioners in computational fluid dynamics, porous media, and flow control who need fast surrogates for high-resolution PDE inference will find the benchmark results and per-epoch timings directly relevant. Researchers benchmarking new operator architectures should also note the FNO⁺ baseline refinements reported here, and readers interested in ablation methodology will find the depth, SE, dilation, and width studies useful.

Authors’ abstract

Fast and accurate surrogates for physics-driven partial differential equations (PDEs) are essential in fields such as aerodynamics, porous media design, and flow control. However, many transformer-based models and existing neural operators remain parameter-heavy, resulting in costly training and sluggish deployment. We propose D-SENO (Dilated Squeeze-Excitation Neural Operator), a lightweight operator learning framework for efficiently solving a wide range of PDEs, including airfoil potential flow, Darcy flow in porous media, pipe Poiseuille flow, and incompressible Navier Stokes vortical fields. D-SENO combines dilated convolution (DC) blocks with squeeze-and-excitation (SE) modules to jointly capture wide receptive fields and dynamics alongside channel-wise attention, enabling both accurate and efficient PDE inference. Carefully chosen dilation rates allow the receptive field to focus on critical regions, effectively modeling long-range physical dependencies. Meanwhile, the SE modules adaptively recalibrate feature channels to emphasize dynamically relevant scales. Our model achieves training speed of up to approximately $20\times$ faster than standard transformer-based models and neural operators, while also surpassing (or matching) them in accuracy across multiple PDE benchmarks. Ablation studies show that removing the SE modules leads to a slight drop in performance.