A Bregman Proximal Viewpoint on Neural Operators

AI summary

Overview

Research area: Machine learning theory for operator learning — specifically neural operators that map between function spaces, with connections to convex analysis, Bregman divergences, and proximal optimization.

Technical level: Advanced. The paper assumes familiarity with convex analysis (subdifferentials, Γ₀ spaces, Fenchel conjugates), Banach and Lebesgue function spaces, Bregman divergences, and proximity operators, and it defines operator layers as solutions of variational problems over function spaces.

Scope: The paper reframes the action of a neural operator layer as the minimizer of a Bregman-regularized functional optimization problem, recovers classical neural operators as a special case, introduces a new "Bregman neural operator" variant, and provides a preliminary universal approximation result plus numerical experiments (arXiv:2610.20120v1 [cs.LG], 14 Oct 2026, CC BY 4.0).

What This Paper Is About

Neural operators learn mappings between function spaces — for example, from an initial condition of a partial differential equation (PDE) to its solution at a later time — and are usually built by analogy with standard neural networks, replacing finite-dimensional linear layers with integral operators and treating activations as pointwise nonlinear maps. This paper takes a different route: instead of writing down the layer as a composed formula, it defines each operator layer as the solution of a regularized optimization problem over functions, where the regularizer determines what the activation operator is. The goal is both to unify existing neural operator architectures under one optimization-based principle and to derive a new architecture from a more general choice of regularizer.

Key Contributions

1. A Bregman proximal reformulation of operator layers. The paper defines each layer ℒ_t : 𝒱_{t−1} → 𝒱_t as the minimizer of −⟨w, 𝒦_t(v) + b_t⟩ + g_t(w) + D_{Φ_t}(w, ℳ_t v), and shows this minimizer equals prox^{Φ_t}_{g_t}(∇̃Φ_t(ℳ_t v) + 𝒦_t(v) + b_t). The activation operator is thereby interpreted as a Bregman proximity operator mapping from dual to primal space.

2. Recovery of classical neural operators (Proposition 3.4). When the Bregman distance reduces to the Euclidean distance — i.e., φ_t = (1/2)|·|² and g_t = Ψ_t − (1/2)‖·‖² — the layer becomes ∇Ψ*_t(ℳ_t v + 𝒦_t(v) + b_t), where ∇Ψ*_t = (ψ*)′(·) matches a range of monotone activation operators. This places standard neural operators, including FNO-style layers, inside the framework.

3. A new architecture: Bregman neural operators (Proposition 3.5). With a general p-uniformly convex Legendre function (≠ |·|²/2) and g_t = 0, the layer takes the form σ(σ⁻¹(v_t) + K_t v + 𝒦_t^ac(v) + b_t), which adds an inverse-activation term to the classical layer. The paper states this variant has the same expressivity as standard neural operators, and that when the weights are zero it reduces to the identity operator.

4. A preliminary universal approximation result and an experimental study. Universal approximation is proven for sigmoidal-type activation operators, and numerical experiments compare the Bregman variant against FNO improvements.

Main Findings

• Layer as a minimization problem. Operator layers are shown to be minimizers of a functional objective whose terms map directly onto architectural parts: the linear/kernel term 𝒦_t(v) + b_t, the regularizer g_t(w), and the Bregman distance D_{Φ_t}(w, ℳ_t v). The general framework permits coupled activations of the form σ₂(σ₁⁻¹(v) + 𝒦_t(v) + b_t), with σ₁ strictly monotone and σ₂ monotone; classical layers arise when σ₁ = Id, and Bregman layers when σ₁ = σ₂ is strictly monotone.

• Table 1 links Legendre functions to known activations. Five (ψ, ψ*′) pairings are reported: ISRU, Sigmoid, tanh, sin, and SoftPlus_β. For example, the sigmoid arises from ψ(t) = t log t + (1−t) log(1−t) on [0,1], and tanh from ψ(t) = log(1−t²) + t·arctanh(t) on [−1,1]. The characterization of SoftPlus is proven in the appendix.

• Scope of the activation correspondence. The authors state that ∇Ψ*_t can only match monotone activation operators, which notably discards GeLU and swish. More precisely, the framework covers any increasing 1-Lipschitzian function, a result they attribute to Proposition 2.3 of Combettes and Pesquet (2020b).

• Identity initialization property. In the Bregman layer, when K_t, 𝒦_t^ac and b_t are zero and ℳ_t is the identity, the layer reduces to the identity operator. The paper reports that this property allows training deeper and more accurate models.

• Fourier layers as a special case. On the unit torus 𝕋^d with translation-invariant kernels, the absolutely continuous part satisfies 𝒦_t^ac(v)(x) = ℱ⁻¹(R_t · ℱ(v))(x), giving the Bregman Fourier layer σ(σ⁻¹(v) + W_t v + ℱ⁻¹(R_t · ℱ(v)) + b_t). Omitting the σ⁻¹(v) term recovers the classical Fourier neural operator layer.

• Basis-independence of the spectral parametrization. The framework is described as transparent to how 𝒦_t^ac is parametrized; using the wavelet transform instead of the Fourier transform is stated to retrieve Wavelet Neural Operators and their novel Bregman variant.

• Optimization-theoretic connections. When 𝒱_{t−1} = 𝒱_t and ℳ_t is the identity, the layer equals (∂Φ_t + ∂g_t)⁻¹(∇̃Φ_t − ℬ_t)(v), a Bregman forward-backward operator known from operator splitting methods. The form of the Bregman layer is also identified as a mirror descent step with mirror map ∇̃Ψ_t.

• Universal approximation. Under the assumptions that σ : ℝ → I is a homeomorphism onto an open interval I, is of sigmoidal type with limits 0 at −∞ and 1 at +∞, and that the input/output spaces are chosen among 𝒞(D̄, ℝⁿ), L^p(D, ℝⁿ), W^{m,p}(D, ℝⁿ) and 𝒞(D̄, ℝᵏ), L^p(D, ℝᵏ), the paper reports a preliminary positive universal approximation result for Bregman neural operators.

• Experimental evidence. The abstract and introduction state that numerical experiments support benefits of the Bregman variant of Fourier neural operators for training deeper and more accurate models, and that the variant's predictive performance improves as depth increases. The specific benchmark datasets, dataset sizes, baseline numbers, and quantitative results are not reported in the content provided.

Methodology in Plain English

Rather than defining a neural operator layer as a formula — apply a linear or integral operator, add a bias, then apply an activation pointwise — the authors define it as the answer to an optimization question: given the current hidden function v, find the next hidden function w that minimizes a sum of three things. The first is a linear cost involving the transformed input (the kernel or linear operator and bias). The second is a convex regularizer g_t. The third is a Bregman divergence measuring how far w is from a transformed version of v, where the divergence is built from a convex "Legendre" function Φ_t. Minimizing this objective is exactly the definition of a Bregman proximity operator, so the activation becomes the proximity operator, and the layer's action is that proximity operator applied to a dual-space quantity.

The authors then work out the consequences of two choices. Choosing the Euclidean distance (φ(t) = (1/2)|t|²) makes the proximity operator collapse to the derivative of a conjugate function, which matches standard monotone activations like sigmoid and tanh — this recovers classical neural operators. Choosing a general Legendre function instead leaves an extra term: the inverse activation applied to the input, giving the new Bregman layer. Because the outer activation cancels the inner inverse activation when layers are composed, no explicit inverse-activation implementation is needed. The authors specialize this to the torus with translation-invariant kernels to obtain the Bregman Fourier layer, and then argue a universal approximation property for sigmoidal activations using this variational structure. A small set of numerical comparisons with FNO variants closes the paper.

Why This Matters

Impact on research. The paper shifts neural operator design from an architectural recipe to a variational principle, which connects it to the mature literature on proximal numerical optimization, Bregman divergences, and operator splitting. This makes a large toolkit available for analysis: convergence theory, forward-backward operators, and mirror descent. Because the framework is general enough to recover existing architectures and generate new ones, it also suggests a principled design space for operator layers beyond the FNO/DeepONet families cited in the paper, and it links back to the finite-dimensional analogue of this idea in Frecon et al. (2022).

Real-world applications (as framed in the paper):

• Solving partial differential equations without resolving them repeatedly from scratch, in particular the initial value problem of mapping an initial condition to the solution at a chosen time horizon.
• Simulating complex dynamical systems.
• Inverse problems and coefficient estimation, the setting targeted by Neural Inverse Operators.
• Physics-informed modeling, which the paper discusses through variants such as PI-DeepONet, LTI-PI-DeepONet, PINO, V-DeepONet, and Modified DeepONet.

Industry relevance. Operator learning is used to build fast surrogate models where repeated high-fidelity simulation is expensive. The claim that the Bregman variant trains deeper and more accurate models while behaving as the identity when weights are zero speaks directly to the training stability and depth-scaling problems that limit current operator architectures, which is where industrial adoption tends to stall.

Future Directions

• Extend the approximation theory. The universal approximation result is described as preliminary and is proven only for sigmoidal-type activations; extending it to the broader class of activation operators allowed by the framework is an open step.
• Derive rates and stability guarantees. The authors motivate their approach by the prospect of transferring results from proximal optimization to neural operators, in the spirit of Combettes and Pesquet (2020b, 2020a); approximation rates, convergence, and robustness guarantees remain to be developed.
• Explore other (Φ_t, g_t) pairs. The paper notes that any pair admitting a closed-form Bregman proximity operator defines a new layer type, and points to Examples 2.9–2.12 in Nguyen (2017) for candidates that have not yet been instantiated.
• Instantiate the framework with other transforms and kernels. Wavelet-based layers (a Bregman WNO) and other bases are proposed as straightforward extensions, as are graph kernel networks and multipole variants, since the framework is stated to be agnostic to how the non-local operator is parametrized.

Target Audience

Researchers in machine learning theory and scientific computing who work on operator learning, PDE surrogates, or the mathematical foundations of deep architectures. The paper is written for readers comfortable with convex analysis and functional analysis; readers wanting only practical guidance will find the architecture descriptions and Table 1 accessible, but the derivations and the universal approximation section require substantial mathematical background. Practitioners building PDE surrogates may also benefit from the concrete Bregman FNO layer formula, with the caveat that the quantitative experimental details are not contained in the provided content.

Authors’ abstract

We present several advances on neural operators by viewing the action of operator layers as the minimizers of Bregman regularized optimization problems over Banach function spaces. The proposed framework allows interpreting the activation operators as Bregman proximity operators from dual to primal space. This novel viewpoint is general enough to recover classical neural operators as well as a new variant, coined Bregman neural operators, which includes the inverse activation operator and features the same expressivity of standard neural operators. Numerical experiments support the added benefits of the Bregman variant of Fourier neural operators for training deeper and more accurate models.