Stronger Normalization-Free Transformers

AI summary

Overview

Research area: Deep learning architecture design — specifically, normalization-free Transformer architectures.

Technical level: Intermediate. The core idea is simple (swap LayerNorm for a scalar function), but the paper relies on familiarity with Transformer blocks, LayerNorm, and training dynamics.

Scope: A systematic study of what makes a point-wise function a viable replacement for normalization layers, culminating in a new function called Dynamic erf (Derf) that outperforms LayerNorm, RMSNorm, and DyT across vision, speech, DNA, and language tasks.

What This Paper Is About

Normalization layers such as LayerNorm, BatchNorm, and RMSNorm have long been treated as essential to training deep Transformer networks, but they rely on activation statistics, which adds memory access and synchronization overhead and makes them sensitive to batch size. The recent Dynamic Tanh (DyT) showed a simple point-wise function can match normalization performance. This paper asks whether a point-wise function can actually surpass normalization layers, and it answers that question by systematically studying the functional properties that matter, searching over a large candidate set, and proposing Dynamic erf (Derf).

Key Contributions

1. A property-level analysis of point-wise functions. The paper isolates four properties — zero-centeredness, boundedness, center sensitivity, and monotonicity — and measures each one's effect through controlled experiments on ViT-Base trained on ImageNet-1K.

2. A large-scale empirical function search. Guided by those properties, the authors construct a candidate set from polynomial, rational, exponential, logarithmic, trigonometric, and cumulative distribution function (CDF) forms, then evaluate them on ViT-Base classification and DiT-B/4 and DiT-L/4 generation.

3. Dynamic erf (Derf). Defined as Derf(x) = γ ∗ erf(αx + s) + β, where erf(x) is the rescaled Gaussian cumulative distribution function, α and s are learnable scalars, and γ and β are learnable per-channel vectors. Derf emerges as the strongest function in the search.

4. A generalization-versus-fitting diagnosis. By measuring evaluation-mode training loss after optimization, the authors show Derf's gains come from better generalization rather than stronger fitting capacity.

Main Findings

• Derf beats LayerNorm and DyT on vision classification. ViT-Base top-1 accuracy on ImageNet-1K: LayerNorm 82.3%, DyT 82.5%, Derf 82.8% (up 0.5% over LN, 0.3% over DyT). ViT-Large: LayerNorm 83.1%, DyT 83.6%, Derf 83.8% (up 0.7% over LN, 0.2% over DyT).

• Derf improves image generation FID. Lower is better. DiT-B/4: LayerNorm 64.93, DyT 63.94, Derf 63.23. DiT-L/4: LayerNorm 45.91, DyT 45.66, Derf 43.94. DiT-XL/2: LayerNorm 19.94, DyT 20.83, Derf 18.92.

• Derf improves speech representation learning. wav2vec 2.0 Base validation loss on LibriSpeech: LayerNorm 1.95, DyT 1.95, Derf 1.93. wav2vec 2.0 Large: LayerNorm 1.92, DyT 1.91, Derf 1.90.

• Derf improves DNA sequence modeling. Mean accuracy over GenomicBenchmarks subtasks: Hyena 85.2% (Norm 85.2%, DyT 85.2%, Derf 85.7%; up 0.5% over both). Caduceus 87.3% (Norm 86.9%, DyT 86.9%, Derf 87.3%; up 0.4% over both).

• Language modeling is roughly a tie with LayerNorm. GPT-2 (124M) on OpenWebText validation loss: LayerNorm 2.94, DyT 2.97, Derf 2.94 (0.00 versus LN, down 0.03 versus DyT).

• Zero-centeredness is required for stable convergence. Performance stays roughly comparable to the zero-centered base function for horizontal shifts with |λ| ≤ 0.5, degrades as the shift grows, and training diverges when |λ| ≥ 2 for both horizontal and vertical shifts.

• Boundedness helps. Clipped versions of unbounded functions (arcsinh(x), logsign(x)) consistently outperform their unclipped baselines across all tested clipping values; unbounded linear(x) fails to converge at all. Gradually blending bounded functions toward a linear form reduces accuracy.

• There is an upper bound on acceptable growth rate. Among inherently unbounded functions, logquad(x) is the fastest-growing function that still converges (82.1%), while faster-growing functions such as linear(x) and power23(x) cause training failure.

• Center sensitivity matters. Introducing a flat zero region around the origin hurts performance; the best result is at λ = 0, degradation becomes obvious once λ exceeds 1.0, and training diverges at λ ≥ 3.0.

• Monotonicity matters. Monotonic functions outperform non-monotonic ones: sin(x) 81.6%, dampx(x) 80.7%, dampexp(x) 81.2%, versus 82.6% for erf(x). Negating a function also costs a small amount (erf(x) 82.6% versus its negated version 82.5%).

• The learnable shift s helps. Adding s improves both top-1 accuracy and FID for every tested function — for example, erf(x) goes from 82.6% to 82.8% and FID from 63.39 to 63.23; tanh(x) goes from 82.5% to 82.6% and FID from 63.94 to 63.71.

• Scalar versus per-channel s makes no significant difference, so the authors use a scalar s for efficiency and simplicity.

• Gains come from generalization, not fitting. Evaluation-mode training loss follows the ordering Norm < Derf < DyT across every tested architecture, including ViT-B (0.2623 / 0.2681 / 0.2714), ViT-L (0.2034 / 0.2066 / 0.2083), DiT-XL (0.1432 / 0.1436 / 0.1440), Caduceus (0.8917 / 0.9129 / 0.9203), and GPT-2 (2.9478 / 2.9702 / 2.9822).

Methodology in Plain English

The authors start from a simple substitution: everywhere a Transformer has a normalization layer, put a point-wise function of the form y = γ · f(αx) + β instead, where f is a fixed base function and α, γ, β are learnable.

They then test one property at a time. To study zero-centeredness, they shift the function horizontally and vertically by controlled amounts (λ ∈ {±1/2, ±1, ±2}). To study boundedness, they clamp inherently unbounded functions (e.g., arcsinh) and, separately, blend bounded functions toward a linear form. To study center sensitivity, they carve a flat zero region of width λ around the origin. To study monotonicity, they compare increasing functions, negated decreasing versions, and non-monotonic hump-shaped and oscillatory functions rescaled to the same output range so the comparison is fair. All of this is run on ViT-Base, with ImageNet-1K top-1 accuracy as the metric.

With those design principles in hand, they assemble a candidate pool from common scalar functions and CDFs, applying transformations such as clipping, scaling, and mirroring, and instantiate each one as y = γ ∗ f(αx + s) + β. They benchmark every candidate on ViT-Base classification and DiT-B/4 and DiT-L/4 generation, keeping erf(x) as the winner, which they then wrap up as Derf.

Derf replaces every normalization layer in the model in a one-to-one manner — pre-attention, pre-FFN, and the final normalization. Initialization is γ = 1, β = 0, α = 0.5, s = 0. The final evaluation spans ViT-Base and ViT-Large, three DiT sizes, two wav2vec 2.0 models, HyenaDNA and Caduceus, and GPT-2 (124M).

Why This Matters

Impact on research. The paper shifts the normalization-free conversation from "can a point-wise function match normalization?" to "which point-wise function should we use, and why?" It supplies a design vocabulary — zero-centeredness, boundedness, center sensitivity, monotonicity — plus a growth-rate ceiling, and it reframes the benefit as an implicit regularizer rather than a fitting improvement.

Real-world applications.

• Training and deploying large vision and diffusion Transformers, where removing statistic-dependent normalization cuts synchronization and memory overhead.
• Speech and audio representation models where batch-size sensitivity of normalization causes training instability.
• Genomic foundation models over long DNA sequences, where per-token statistics are costly.
• On-device or distributed inference, where a stateless scalar mapping is cheaper to implement than a normalization kernel.

Industry relevance. Derf is a drop-in, near-zero-cost change: one extra scalar parameter (s) relative to DyT and no change to the surrounding architecture. That makes it attractive for teams already training or fine-tuning Transformer stacks who want accuracy or FID improvements without redesigning the model.

Future Directions

• Scaling to large language models. The language result is only for GPT-2 (124M) on OpenWebText, where Derf merely matches LayerNorm; whether the vision and generation gains transfer to modern large-scale LLM pretraining is untested in the reported content.
• Theory for why bounded point-wise functions generalize better. The paper offers an intuitive explanation based on limited adaptability acting as an implicit regularizer, but a formal account is not provided.
• Extending and formalizing the function search. The current search is empirical over a hand-constructed candidate set; automated search, or principled extensions of the candidate space beyond the forms listed, remains open.
• Approximating Derf. Section 7 mentions "an analysis of an approximation of Derf," but the truncated content does not report those results; whether Derf can be replaced by a cheaper or hardware-friendlier approximation is an open question.

Target Audience

Machine learning researchers working on Transformer architecture and training stability; practitioners who train vision, diffusion, speech, or genomic models and want a low-cost replacement for normalization layers; and engineers interested in the efficiency tradeoffs of statistics-free operators. Readers who already know LayerNorm and the DyT result will get the most out of the property analysis and the cross-domain benchmark tables.

Authors’ abstract

Although normalization layers have long been viewed as indispensable components of deep learning architectures, the recent introduction of Dynamic Tanh (DyT) has demonstrated that alternatives are possible. The point-wise function DyT constrains extreme values for stable convergence and reaches normalization-level performance; this work seeks further for function designs that can surpass it. We first study how the intrinsic properties of point-wise functions influence training and performance. Building on these findings, we conduct a large-scale search for a more effective function design. Through this exploration, we introduce $\mathrm{Derf}(x) = \mathrm{erf}(αx + s)$, where $\mathrm{erf}(x)$ is the rescaled Gaussian cumulative distribution function, and identify it as the most performant design. Derf outperforms LayerNorm, RMSNorm, and DyT across a wide range of domains, including visual recognition and generation, speech representation, and DNA sequence modeling. Our analysis also suggests that the performance gains of Derf largely stem from its improved generalization rather than stronger fitting capacity. Its simplicity and stronger performance make Derf a practical choice for normalization-free Transformer architectures.