Smoothness Errors in Dynamics Models and How to Avoid Them

AI summary

Overview

Research area: Machine learning for partial differential equations (PDEs) and geometric deep learning — specifically graph neural networks and mesh-based models for dynamics prediction.

Technical level: Advanced. The paper combines spectral graph theory (Rayleigh quotient), group-invariant approximation theory over SU(n), and discrete differential geometry (cotangent and Robust Laplacians on meshes) with empirical benchmarks on PDE rollouts.

Scope: The paper argues that exactly smoothness-preserving (unitary) graph convolutions are over-constrained for physical dynamics, and proposes relaxed variants, generalized from graphs to meshes, that outperform strong mesh-aware baselines on PDE and weather-forecasting tasks.

What This Paper Is About

Graph neural networks used to solve PDEs suffer from oversmoothing, where neighboring node features become increasingly similar. Unitary convolutions were proposed to fix this by mathematically preserving the Rayleigh quotient — a measure of signal smoothness — but many physical systems, such as diffusion, naturally become smoother over time, so forcing smoothness to stay constant can be the wrong constraint. The paper characterizes when unitary models are too restrictive, then introduces relaxations that let a model match the true smoothness of the target dynamics rather than freezing it.

Key Contributions

1. An approximation-error lower bound for unitary models (Section 4). The authors prove (Theorem 1) that the regression error of any unitary function is bounded below by the variance of the norm of the target function, averaged over concentric spheres in the data domain. This shows unitary models are over-constrained when the norm of the ground-truth function has high angular dependence.

2. Relaxed unitary convolutions (Section 5.1–5.2). Two relaxations are introduced: Taylor truncation of the Lie unitary convolution (yielding R-UniGraph) and a compositional smoothness-breaking architecture in which a smoothness-preserving block built from unitary layers is followed by a smoothness-breaking block such as an MLP or GCN.

3. Generalization of the Rayleigh quotient and unitary convolution from graphs to meshes (Section 5.3). The authors define a mesh Rayleigh quotient (Definition 4) using the Robust Laplacian (Sharp and Crane, 2020) with cotangent weights, and define separable and Lie unitary mesh convolutions. Corollary 1 states that these preserve the mesh Rayleigh quotient under the assumption that the mesh satisfies the intrinsic Delaunay criterion (Bobenko and Springborn, 2007). The resulting relaxed architecture is named R-UniMesh.

4. A systematic empirical study of smoothness behavior in GNN dynamics models (Section 6). The paper measures how GCNs, unitary models, mesh-aware transformers, and equivariant networks behave in terms of the Rayleigh quotient, finding that approximately preserving smoothness is a key inductive bias — matching or exceeding the benefit of equivariance.

Main Findings

• Unitary convolutions are provably over-constrained for some dynamics. Theorem 1 shows the error lower bound for a unitary model is the averaged variance of the norm of the target function across concentric spheres; when the target's norm varies strongly with angle, this bound is large.

• GCNs oversmooth relative to the target, not just relative to the input. In the heat-flow experiment, the authors report that GCNs smooth beyond what the target signal requires, while the relaxed model (R-UniGraph) is often initialized undersmooth and learns to increase smoothness during training to match the target.

• Relaxed models win on heat flow over grid graphs. On 10,000 two-dimensional grids with 20 randomly placed heat sources each, predicting the heat field at t = 4 from t = 3, R-UniGraph achieved MSE 0.11·10⁻² and mean Rayleigh quotient error (MRE) 2.07·10⁻² with truncation T_max = 3, versus GCN at 1.08·10⁻² MSE / 5.99·10⁻² MRE and a Lie unitary model at 0.14·10⁻² MSE / 8.86·10⁻² MRE. Best of 5 runs is reported.

• T_max = 10 is sufficient to preserve the Rayleigh quotient. A sensitivity analysis of Taylor truncation found that truncating at 10 terms preserves smoothness, consistent with prior work (Kiani et al., 2024).

• Qualitative behavior on the armadillo mesh. At rollout timestep T = 190 of the heat equation, the paper reports that R-UniMesh tracks the ground truth, EMAN oversmoothes, and Hermes undersmoothes.

• Strong results on mesh PDE benchmarks. On heat (α = 1), wave (c = 1), and Cahn-Hilliard equations over PyVista test meshes from Park et al. (2023), R-UniMesh is reported as competitive across all tasks and especially strong on the heat equation. Its reported values are: heat NRMSE 51.9 ± 3.6, SMAPE 79.7 ± 5.6, RE 9.1 ± 7.4; wave NRMSE 236.5 ± 6.4, SMAPE 385.2 ± 1.2, RE 93.5 ± 25.4; Cahn-Hilliard NRMSE 123.9 ± 2.6, SMAPE 167.3 ± 10.6, RE 18.9 ± 10.4. Errors in the table are scaled by ×196, the number of rollout timesteps, and cells with a dash indicate models that did not converge for a given metric.

• Smoothness control rivals or exceeds other geometric inductive biases. The authors state that their mechanism for approximately preserving smoothness provides equal or greater improvement than other inductive biases such as equivariance, and that geometric inductive biases matter most on unseen meshes with complex geometry.

Methodology in Plain English

The starting point is a standard trick in graph learning: measure how "smooth" a signal is on a graph using the Rayleigh quotient, which compares the differences between neighboring node features against the total signal magnitude. A perfectly uniform signal scores zero. The paper defines oversmoothing and undersmoothing relative to the target signal: a model oversmoothes if its output is smoother than the ground truth, and undersmoothes if it is rougher.

Prior work built unitary convolutions — layers whose transformations are rotations, preserving vector lengths — to guarantee the Rayleigh quotient never changes across layers. The authors instead truncate the matrix exponential in the Lie unitary convolution after a small number of Taylor terms, so the layer only approximately preserves smoothness, leaving room for the extra smoothing that diffusive physics requires. This single knob is T_max, and the paper shows the results are stable across values of it (using T_max = 3 for the grid-graph experiment).

Because unitary layers cannot change a signal's channel dimension, the second relaxation instead zero-pads the input features to a chosen hidden width — padding preserves norms, and therefore the Rayleigh quotient — runs the signal through several unitary layers, and then passes it through an ordinary unconstrained network (an MLP or GCN). This raises parameter count without increasing depth, which would make training unstable.

To move from graphs to meshes, the authors replace the graph Laplacian with the symmetric cotangent Laplacian, which approximates the Laplace-Beltrami operator of the underlying surface. Because cotangent weights can be negative — which would break the Rayleigh quotient as a smoothness measure — they use the Robust Laplacian, which minimally rewires edges so the intrinsic Delaunay criterion holds and off-diagonal weights are nonnegative. They then show that unitary convolution with these weights preserves the mesh Rayleigh quotient.

Finally, they train and compare these models on simulated heat diffusion, three PDE systems on complex 3D meshes, and weather forecasting on the Earth mesh from WeatherBench 2, benchmarking against GCNs, MPNNs, mesh transformers, GemCNN, EMAN, Hermes, and an EGNN-style Euclidean equivariant model.

Why This Matters

Impact on research: The paper reframes oversmoothing as a relative problem — the goal is not to preserve smoothness at all costs but to match the smoothness of the target dynamics. It also introduces architectural inductive biases designed to match the Rayleigh quotient of labeled graphs, and it is among the first to use cotangent edge weights from the Robust Laplacian in deep dynamics models. Crucially, it

Authors’ abstract

Modern neural networks have shown promise for solving partial differential equations over surfaces, often by discretizing the surface as a mesh and learning with a mesh-aware graph neural network. However, graph neural networks suffer from oversmoothing, where a node's features become increasingly similar to those of its neighbors. Unitary graph convolutions, which are mathematically constrained to preserve smoothness, have been proposed to address this issue. Despite this, in many physical systems, such as diffusion processes, smoothness naturally increases and unitarity may be overconstraining. In this paper, we systematically study the smoothing effects of different GNNs for dynamics modeling and prove that unitary convolutions hurt performance for such tasks. We propose relaxed unitary convolutions that balance smoothness preservation with the natural smoothing required for physical systems. We also generalize unitary and relaxed unitary convolutions from graphs to meshes. In experiments on PDEs such as the heat and wave equations over complex meshes and on weather forecasting, we find that our method outperforms several strong baselines, including mesh-aware transformers and equivariant neural networks.