Language as a Wave Phenomenon: Semantic Phase Locking and Interference in Neural Networks

AI summary

Overview

Research area: Complex-valued deep learning, spectral sequence models, and mechanistic interpretability of neural sequence architectures. The paper sits at the intersection of signal processing (Fourier methods, phase, interference) and neural sequence modeling.

Technical level: Advanced. The paper assumes familiarity with FFT/IFFT pipelines, complex algebra, Transformer attention, unitary constraints, and causal/interventional ablation methodology.

Scope (one sentence): Using a purpose-built complex-valued architecture called PRISM, the authors test whether phase — rather than activation magnitude — can serve as the primary carrier of task-relevant information in neural sequence models, evaluating on WMT14 De-En probing experiments and WikiText-103 masked reconstruction with interventional ablations.

What This Paper Is About

In standard real-valued Transformers, semantic importance is tied to activation magnitude: signals are emphasized or suppressed by changing their scale. The authors ask whether that coupling is necessary, or whether information can instead be carried in the phase of a complex-valued representation, with interactions occurring through rotation and interference rather than additive accumulation. To study this, they build PRISM (the Phase-Rotating Interference Spectral Model), which replaces attention with an FFT → learned complex filter → IFFT pipeline under a strict unit-norm constraint (|z| = 1), and then test causally whether the phase of the learned filters actually carries the model's working information.

Key Contributions

1. PRISM, a constrained complex-valued spectral architecture. A sequence model built from four phase-aware components: Rotary Semantic Embeddings (RoSE), which rotate token content vectors multiplicatively by positional phase instead of adding a positional encoding; Gated Harmonic Convolutions, which replace attention with an FFT → H → IFFT pipeline at O(N log N) and apply Cartesian gating to the real and imaginary components independently so the network can alter phase angles; ModReLU, a non-linearity that rectifies magnitude while preserving phase; and Phase-Preserving Layer Normalization plus Phase-Preserving Dropout, which avoid destroying the phase angle during optimization.

2. A hybrid architecture family combining phase routing with attention. The paper introduces the PRISM Hybrid, the Hybrid Spectral Sequence Model (HSSM), and the Wave-Particle Transformer (WPT), which run a phase-coded relational stream in parallel with a real-valued token-identity stream and fuse them — testing whether phase and attention are complementary rather than competing.

3. Three hypothesis-driven probes of phase structure on WMT14 De-En. Phase coherence, phase rotation under ambiguity, and a minimum-sequence-length ("spectral density") threshold were tested as qualitative predictions of phase-based semantic coding.

4. Interventional ablations isolating phase as the information carrier. Filter-level interventions (masking, reversing, shuffling phase) and layer-wise causal interventions on the encoder state directly test whether the learned spectral pathway depends on phase, with the constrained and unconstrained models compared side by side.

Main Findings

• Phase-only filters preserve nearly all performance. In the FFT → H → IFFT pipeline, keeping only the phase of the learned filter (H/|H|) retains more than 95% of performance, with only 1.41× degradation in the constrained model and 1.51× in the unconstrained model. Removing the filter entirely degrades by 328× (constrained) and 78× (unconstrained); keeping only magnitude degrades by 153× and 173×; reversing phase by 245× and 139×; shuffling phase by 160× and 161×.

• Layer-wise interventions show the same asymmetry. Scrambling phase at any single layer of the unconstrained complex PRISM encoder causes 72–205× degradation, while scrambling magnitude causes only 1.1×. The constrained model shows even stronger phase dependence (approximately 2000×).

• Phase coherence tracks semantic relatedness, not similarity. Across 107 deduplicated single-token word pairs, synonyms had mean phase coherence R = 0.197 ± 0.096 (95% CI [0.172, 0.227]), antonyms R = 0.133 ± 0.064 (CI [0.114, 0.155]), and random pairs R = 0.071 ± 0.029 (CI [0.059, 0.082]). All pairwise differences were significant (Mann-Whitney U with Bonferroni correction; permutation test p < 10⁻⁴; Synonym vs. Random p < 10⁻⁷; Antonym vs. Random p = 2.2 × 10⁻⁴). Antonyms exceeding random pairs suggests coherence reflects topical relatedness rather than meaning overlap.

• Polysemous tokens undergo larger phase rotations. Across 146 single tokens, polysemous tokens showed a heavy-tailed rotation distribution (skewness γ₁ = 1.59) concentrated at Layer 3, compared to γ₁ = 0.91 for unambiguous tokens.

• A minimum sequence length is required for phase-based computation. At L = 1 and L = 2 the model produces repetitive loops regardless of input semantics, because a single token yields only a DC component under FFT and a pair yields only two frequency bins — insufficient spectral resolution for the harmonic filters.

• The WPT is the strongest architecture and beats the Transformer baseline. At L = 4096 on WikiText-103, the WPT achieved 4.94 perplexity (Top-1 67.2, Top-5 82.8) versus 5.28 for the Transformer (Top-1 66.1, Top-5 81.9), while using 18% fewer core parameters (12.9M vs. 15.8M). The PRISM Hybrid reached 6.06 PPL and the HSSM 6.47 PPL; the real-valued FNet Hybrid collapsed to 9.87 PPL.

• The real-versus-complex gap is attributable to phase coding in the relational stream. Replacing the WPT's PRISM stream with a parameter-matched real spectral encoder (13.5M core params vs. 12.9M) yields 5.55 PPL versus 4.94 — a 12.3% gap. A topology-matched real encoder using rfft/irfft with a d → 2d → d expand-compress design reached 6.15 PPL despite more core parameters than the complex PRISM variant at 5.80.

• Phase-based computation is emergent, not prescribed. Replacing all phase-preserving components with phase-destroying alternatives (CReLU + standard LayerNorm) improved standalone performance to 5.80 PPL (versus the constrained PRISM's 6.06), yet the model still organized its computation around phase. In hybrid settings, however, CReLU-WPT scored 5.46 PPL, worse than the constrained complex WPT at 4.94 — the authors hypothesize that CReLU's quadrant routing can suppress noise independently, reducing the pressure to align phase angles precisely, and that ModReLU's constraint forces the phase stream to specialize in relational interference once attention handles token selection.

• Throughput scaling favors PRISM under controlled comparison. On an A100 at batch=2, under BF16 autocast with Flash Attention, the Transformer ran at 237,248 tokens/s at L = 4096 versus PRISM at 118,793 — but the paper attributes much of this to framework support (complex FFT/IFFT remains in FP32 under BF16 autocast). Under FP32 parity, PRISM reached 91,602 tokens/s at L = 4096 (versus the Transformer's 90,486) and retained 77% of its throughput from L = 4096 to L = 16384, while the Transformer retained 35%. With Flash Attention disabled, the Transformer and WPT ran out of memory at L = 8192 while PRISM continued at 72,954 tokens/s.

• Parameter efficiency in the encoder. The PRISM encoder requires 31.2% fewer parameters than the standard Transformer encoder (13.0M vs. 18.9M) in the WMT14 configuration. On WMT14 De-En, the Static RoSE configuration reached 0.799 COMET, trailing FNet (0.805) and the Transformer (0.821). Uniformly attenuating spectral filter weights (α ≈ 0.74) degraded COMET by only 3%, consistent with phase invariance to scalar scaling.

Methodology in Plain English

The authors proceed in two stages rather than trying to win a benchmark outright.

Stage one — hypothesis-guided probing on WMT14 De-En. They build a deliberately restricted version of PRISM that uses fixed positional phase rotations without content-dependent steering, and compare it against a standard Transformer with Rotary Position Embeddings (the "rate-coding" control) and FNet (the "mathematically isomorphic" spectral control, whose encoder depth they raise to 7 layers versus 6 for PRISM so the baseline is not under-parameterized). They tokenize with the Helsinki-NLP OPUS-MT model, filter sequences longer than 128 tokens (discarding under 1% of data), and use dynamic bucketing with width 4 and target size 20,000. All models share identical hyperparameters (AdamW, weight decay 0.01, cosine schedule with 600 warmup steps), and the peak learning rate is capped at 6 × 10⁻⁴ because the Transformer control destabilized above that, even though PRISM remained stable at 8 × 10⁻⁴. Everything runs in FP32 to avoid catastrophic cancellation of phase information. They then instrument each encoder layer with forward hooks and measure four quantities: signal gain (output/input magnitude ratio), phase rotation (angular displacement between input and output phasors), rotation skewness (whether the layer rotates all tokens uniformly or only a subset), and phase coherence R (a complex generalization of cosine similarity, computed as the magnitude-weighted mean resultant length of phase differences across frequency bands).

Stage two — scale and causal testing on WikiText-103. At sequence length 4096 with a BERT-style masked language modeling objective and dynamic masking, they compare five architectures standardized to roughly a 33M parameter budget and a 32k BPE vocabulary: a standard Transformer with RoPE, an FNet Hybrid with real-valued 2D Fourier mixing, the PRISM Hybrid (a full-width d = 512, 5-layer PRISM encoder with a phase bridge and a 1-layer Transformer refiner), the HSSM (parallel 9-layer FNet and 9-layer PRISM streams at d = 256), and the WPT (a 6-layer Transformer stream at d = 256 alongside a 6-layer phase-coded relational stream, fused by concatenation and a 1-layer attentive refiner). Training runs for 40 epochs with global batch size 32 (8 physical × 4 gradient accumulation steps), cosine schedule peaking at 1 × 10⁻³, 10% warmup, and dropout p = 0.1. PRISM variants use RMSNorm and zero weight decay; baselines use standard LayerNorm and weight decay λ = 0.01. To generalize beyond a fixed length, static filters are replaced with "Neural Filters" — a 3-layer MLP with hidden dimension 64 and SiLU activations that maps sinusoidal position encodings to complex filter weights — and Dynamic RoSE adds a content-dependent phase shift alongside the positional rotation.

The decisive move is the ablation design: rather than only comparing architectures, they freeze trained checkpoints and surgically alter a single variable in the spectral filter (remove it, keep magnitude only, keep phase only, reverse phase, shuffle phase), then repeat at the level of individual encoder layers by scrambling phase or magnitude in the state itself.

Why This Matters

The paper reframes a foundational assumption in sequence modeling: that salience must be encoded as scale. If a learned network reliably routes relational information through phase, then magnitude-based selection is a convention rather than a necessity, and the design space of sequence architectures is wider than the dominant attention-plus-gating template suggests. The result that the complex parameterization provides a useful inductive bias is notable because complex multiplication builds the subtractive cross-term (W_r x_r − W_i x_i) + i(W_r x_i + W_i x_r) directly into the parameterization, rather than requiring a real-valued model to discover it from unconstrained mixing. The paper also extends a classical signal-processing observation (precedent cited to Oppenheim and Lim, 1981) about the dominance of phase in structured representations to learned neural sequence representations.

The finding that phase-based dependence persists even when all phase-preserving design choices are removed is an interpretability result: the constraint shapes how the model computes, but the reliance on phase comes from the complex spectral topology itself.

Potential real-world applications (the paper evaluates only WMT14 translation probing and WikiText-103 masked reconstruction; it does not report deployed systems):

• Long-context sequence modeling, where the measured throughput retention (77% from L = 4096 to L = 16384 under FP32 parity) and the ability to continue running when Transformer variants ran out of memory point toward tasks with very long inputs.
• Parameter-constrained deployment, since the WPT outperformed the Transformer baseline while using 18% fewer core parameters, and the PRISM encoder required 31.2% fewer parameters than the standard Transformer encoder in the WMT14 configuration.
• Machine translation and language modeling backbones, the two task families the paper actually benchmarks.
• Architectures with interchangeable reasoning primitives, where a phase-based relational stream could serve as a drop-in complement to an attention-based token-selection stream.

Industry relevance: The throughput analysis is candid about a real deployment obstacle — complex FFT/IFFT operations remain in FP32 under BF16 autocast because complex BF16 kernels are not available, and Flash Attention gives real-valued attention a fused-kernel advantage. Companies building inference infrastructure at long context lengths would weigh the intrinsic O(N log N) scaling against this current framework gap. The zero-weight-decay finding for the PRISM Hybrid (the unit-norm constraint acting as implicit regularization) and the general parameter efficiency are also directly relevant to teams trading accuracy against model size.

Future Directions

• Closing the framework gap for complex-valued models. The paper explicitly frames the BF16 throughput gap as reflecting framework support rather than intrinsic cost; native complex BF16 kernels and fused spectral operators are the obvious next infrastructure step, given that PRISM already matches the Transformer's FP32 throughput at L = 4096.
• Explaining why CReLU helps standalone but hurts in hybrid settings. The authors present their quadrant-routing hypothesis as a hypothesis rather than a conclusion, and the reversal (CReLU better standalone at 5.80 vs. 6.06, but worse in the W

Authors’ abstract

In standard Transformer architectures, semantic importance is often conflated with activation magnitude, obscuring the geometric structure of latent representations. To disentangle these factors, we introduce PRISM, a complex-valued architecture designed to isolate the computational role of phase. By enforcing a strict unit-norm constraint ($|z| = 1$) and replacing attention with gated harmonic convolutions, the model is encouraged to utilize subtractive interference in the frequency domain to suppress noise, rather than relying on magnitude-based gating. We utilize this constrained regime to study a hybrid architecture -- fusing phase-based routing with standard attention -- which achieves improved parameter efficiency and representation quality compared to baselines in our evaluated settings. Mechanistically, interventional ablations indicate that the model carries substantial task-relevant information in phase: preserving phase largely maintains performance, whereas disrupting phase causes severe degradation. Together, these results suggest that phase-based spectral interference is a usable computational mechanism for neural sequence modeling at the evaluated scale.