On Making $\mathrm{SE}(2)$-Invariant Networks Optimal

AI summary

Overview

Research area: Geometric deep learning / symmetry-aware neural network architectures, specifically building SE(2)-invariant convolutional neural networks (invariance to translations and rotations in the plane) using classical invariant theory and moment invariants.

Technical level: Advanced. The paper draws on group representation theory, classical moment invariants (Hu, Flusser, Zernike), and steerable convolutions, though the neural module itself is presented as plug-and-play.

Scope: The paper derives a provably minimal and discriminative SE(2)-invariant feature map, implements it as a drop-in convolutional module, and benchmarks it against E(2)-equivariant steerable CNNs across eight image datasets plus one newly constructed microscopy dataset.

What This Paper Is About

Rotation is often a nuisance variable: a satellite image of a runway, a histology slide, or a microscopy grain looks semantically the same at any orientation, but a standard CNN gives different answers when the input is rotated. Existing fixes build networks that carry structured, rotation-equivariant feature fields, which are principled but expensive and force every layer (convolutions, nonlinearities, normalization) to be redesigned. This paper instead asks what the optimal rotation-invariant feature map looks like, proves a construction that loses no information beyond the group action and contains no redundant features, and packages it as a module that slots into ordinary CNN pipelines.

Key Contributions

1. An optimal invariant map. Building on classical invariant theory, the authors derive an invariant feature map and prove it is both discriminative (preserving all information up to the SE(2) group action) and minimal (no subcollection of the features is still discriminative).
2. A plug-and-play SE(2)-invariant CNN module. The theory is translated into a moment projection layer plus an invariant layer that projects everything back to type-0 (invariant) features, requiring no modification to standard nonlinearities, normalization layers, or 1×1 convolutions.
3. Competitive accuracy at lower computational cost. Experiments across rotation-symmetric image benchmarks show competitive or better accuracy than steerable equivariant baselines while using models typically two to three times smaller, with lower runtime.
4. A theoretical and empirical analysis of equivariance vs. invariance. The paper examines how the two relate and what each contributes to robustness and flexibility, including a rotation consistency metric that separates the two model classes sharply.

Main Findings

• Accuracy is competitive or better with far fewer parameters. On MNIST-Rot-Test the method reaches 99.42 ± .07 with 119k parameters versus 97.20 ± .14 with 362k for E(2)-CNNs. On RESISC45 it reaches 85.77 ± .47 (230k parameters) versus 84.02 ± .27 (365k), and on EuroSAT rgb 95.29 ± .40 (149k) versus 93.90 ± .34 (363k). On DTD (top-five accuracy) the two are essentially tied at 67.14 ± .27 versus 67.18 ± .81, and on Galaxy Zoo the baseline is higher at 84.39 ± .08 versus 83.30 ± .03.
• Rotation consistency improves dramatically. Using the rotation consistency index (RCI), the L2 distance between logits for an upright input and its rotated counterpart, the method reports values such as 0.01 on MNIST-Rot-Test and rotMNIST, versus 0.89 and 0.64 for E(2)-CNNs. On benchmarks with explicit rotations the paper reports reducing RCI by an order of magnitude. Even where accuracy is slightly lower (Galaxy Zoo), RCI is 0.27 versus 0.82.
• The proposed invariant representation beats the Flusser basis in practice. Table 2 compares magnitude-plus-relative-phase features (Eq. 13) against real and imaginary parts of the Flusser basis: 99.42 ± .07 vs 99.27 ± .05 (MNIST-Rot-Test), 99.08 ± .04 vs 98.77 ± .03 (rotMNIST), 85.77 ± .47 vs 83.33 ± .21 (RESISC45), 95.29 ± .40 vs 93.77 ± .36 (EuroSAT). The Flusser form required extra normalization to avoid numerical instability from large magnitudes.
• Lower computational complexity and measured speedups. Steerable CNNs require type-to-type interactions scaling as O(N_in · N_out), whereas projecting directly to type-0 features scales linearly in the number of moments. Measured forward-pass speedups reach up to 14× for higher moment orders. On NFFA with maximum type k = 3, an epoch takes 227 seconds on a single A100 GPU with e2cnn versus 98 seconds for the proposed method.
• Better sample efficiency in low-data regimes. On the newly constructed NFFA benchmark, a roughly 300k-parameter model outperforms ResNet18 with 11M parameters trained with rotation and flip augmentation, with the largest gap at small training fractions such as 6.2% and 12.5%; the gap narrows as data increases.
• Larger kernels help invariance, but not for the baseline. Larger convolutional kernels improved rotational consistency in the proposed method, while applying larger kernels in e2cnn reduced accuracy and increased cost and parameter count.
• Local invariance retains orientation information. The paper notes that performance stays strong despite enforcing local invariance, indicating relative orientation can still be inferred from invariant features computed over the continuous spatial domain.
• Known degeneracy of the reference-moment construction. The Flusser basis cannot distinguish patterns for which all type-1 moments vanish, such as n-fold symmetric patterns; the flexible basis of Bujack et al. handles this by choosing a nonzero moment as reference but requires a-priori knowledge and keeping multiple reference frames.

Methodology in Plain English

The starting point is the classical idea of complex moments: projecting an image patch onto basis functions shaped as a radial part times a complex exponential in the angle, ψ_{k,l}(r, θ) = R_{k,l}(r) e^{ikθ}. Each moment has a "type" k, and under a rotation by angle θ a moment of type k simply picks up a phase factor e^{ikθ}.

The authors then do two things with that phase behavior. First, the magnitude |m_k| of any moment is already rotation-invariant. Second, the phase differences between moments can be made invariant by measuring each phase relative to a reference moment m_1, the same trick the classical Flusser basis uses to cancel rotational phase. The paper's construction combines these into a real-valued invariant,

φ_O(m_1, m_k) = σ(|m_1|, |m_k|) · cos(θ_k − k θ_1),

where σ is a magnitude-dependent modulation. That modulation matters: phases of very small moments are numerically unstable, and directly forming products like m_k · m_1^(−k) causes runaway magnitudes in deep layers.

In the network, instead of computing invariants from a single whole pattern, the method evaluates them at every spatial location, making the module a form of steerable convolution whose output is projected straight back to invariant type-0 features. The moment kernels use B-spline radial parameterizations (fixed or learned) and larger kernel sizes than standard CNNs to reduce discretization error. Moments are normalized with a shared layer normalization across types, similar to Equiformer; the paper reports performance is not sensitive to the normalization choice. The invariant layer is followed by a 1×1 convolution, normalization, and nonlinearity in a design similar to MBConv blocks, and blocks are stacked stage-wise at decreasing resolution and increasing channel width.

Evaluation covers MNIST-Rot-Test, RotMNIST, RESISC45, EuroSAT, DTD, Galaxy Zoo, Colorectal Histology, and a newly curated NFFA dataset built from NFFA-Europe microscopy images (resized and cropped to 224 × 224). Baselines are E(2)-CNNs via the e2cnn framework, with H-Nets, standard CNNs, and transformer models reported in the appendix. Besides accuracy, the paper introduces and reports the rotation consistency index.

Why This Matters

Impact on research. The paper reframes the cost of symmetry: instead of carrying equivariant feature fields through every layer, it shows that a minimal, discriminative invariant map can be computed once per block, restoring compatibility with off-the-shelf CNN components. It also provides a theoretical treatment of when invariance suffices versus when equivariance is needed, which speaks to ongoing debates about expressivity in geometric deep learning. The distinction between the proposed basis and the classical Flusser basis is a concrete demonstration that theoretically equivalent constructions can differ substantially in numerical behavior.

Real-world applications:

• Remote sensing and Earth observation (RESISC45, EuroSAT): land-use and land-cover classification where scene orientation carries no semantic meaning.
• Scientific imaging and materials characterization (NFFA SEM microscopy): scarce, high-variability data where symmetry priors improve sample efficiency.
• Computational pathology (Colorectal Histology): tissue classification where slides can be scanned at arbitrary orientations.
• Astronomy and geoscience (Galaxy Zoo; Lost Oil Wells is also listed as a dataset considered): image classification where object rotation is arbitrary or physically irrelevant.

Industry relevance. The module is described as plug-and-play into standard CNN pipelines, uses 2–3× fewer parameters than the equivariant baseline, avoids dependence on specialized steerable-convolution frameworks, and delivers up to 14× faster layer forward passes at higher moment orders. That combination matters for deployment on constrained hardware and for training budgets. The sample-efficiency gains in the ~300k-parameter regime are directly relevant to industrial settings where labelled data is scarce.

Future Directions

1. Adaptive selection of moment types — choosing which angular and radial indices to use per layer rather than fixing N_k and N_l.
2. Extensions to broader transformation groups beyond SE(2), and to three-dimensional settings, for example by projecting 3D features through 2D as explored in prior work.
3. Combining invariant and equivariant representations in a single architecture, since the paper's own analysis shows relative orientation is still recoverable from invariant features over the continuous domain.
4. Handling degenerate cases more robustly — the invariant map relies on non-zero moment responses, and features can become unreliable when moment magnitudes are very small; the paper states such cases are rare in practice and can be mitigated by computations over the continuous spatial domain, but this remains an open design question.

Target Audience

Researchers and practitioners in geometric deep learning, equivariant and symmetry-aware network design, and scientific machine learning who want rotation invariance without the cost of full equivariant feature fields. It will also be useful to applied researchers in remote sensing, materials science, microscopy, and computational pathology who need rotation-robust image classifiers under data and compute constraints, and to readers with a background in classical moment invariants or pattern recognition who want to see that theory connected to modern CNN modules. A working familiarity with group representation theory and moment-based invariants is assumed.

Authors’ abstract

Respecting a-priori known symmetries of underlying data is a key principle in designing efficient neural architectures. Convolutional neural networks are a canonical example, encoding translation equivariance. Extending this inductive bias to richer symmetry groups such as $\mathrm{SE}(2)$ in the most effective way possible is an ongoing architectural and computational challenge. Building on classical invariant theory, we derive invariant feature maps that are provably optimal: minimal and information-preserving with respect to the symmetry group. Our construction is designed to integrate seamlessly into standard architectures, requiring no modification to nonlinearities or normalization layers. As a concrete implementation, we introduce a plug-and-play module for convolutional neural networks that enforces $\mathrm{SE}(2)$ invariance, capturing both translations and rotations within standard architectures. The resulting models are both efficient and expressive. Experiments show competitive or improved accuracy compared to state-of-the-art equivariant methods at significantly lower computational cost. Finally, we provide a theoretical and empirical analysis of the relationship between equivariance and invariance, offering insight into their roles in controlling robustness and flexibility.