It's Not RoPE that Creates Sinks: The Role of Self-Concentration and Value-Non-Mixing in Attention

AI summary

Overview

Research area: Mechanistic interpretability of large language models, specifically the internal dynamics of attention layers.

Technical level: Intermediate (requires familiarity with transformer attention, softmax, positional encodings, and quantization concepts).

Scope: An empirical mechanistic study isolating why Attention Sinks and Massive Activations cluster at the first token of a sequence, using causal interventions on five open-weight LLMs.

What This Paper Is About

Large language models consistently dump a disproportionate share of attention onto the first token of a sequence — a phenomenon called "Attention Sink" — and the same position often carries a handful of enormously large activation values ("Massive Activations"). These two phenomena co-occur and make low-precision quantization difficult, but it has never been clear which property of the first position actually causes them. The authors test three candidate explanations — the BOS token, the first positional encoding (RoPE), and the forced self-attention imposed by causal masking — and find that the last one, together with the resulting "Value-non-mixing," is the real driver.

Key Contributions

1. Disentangling the causes of initial-position AS and MAs. The authors separate the BOS token's identity, RoPE positional encoding, and causal-mask-induced self-concentration, and show through targeted interventions that each contributes differently — with self-concentration emerging as the dominant structural factor.

2. Introducing "Value-non-mixing" as a mechanistic explanation. They formalize the state in which a query's attention output is composed solely of its own Value vector rather than a weighted blend of multiple Value vectors, and argue this state — not self-concentration per se — is what triggers AS and MAs.

3. Causal intervention evidence across five models. Forcing self-concentration at an arbitrary non-initial position induces AS and MAs there in all five models tested, while perturbing RoPE indices for the first token does almost nothing.

4. A controlled repeated-token experiment. By feeding sequences of identical tokens (with a single distinct token inserted at various positions), the authors reproduce Value-non-mixing without any attention surgery, and show AS and MAs appear exactly over the span of positions where the state holds.

Main Findings

• The BOS token is not the whole story. Moving the BOS token to position 16 leaves AS at its new location, but AS and MAs still appear at position 1 even when a non-BOS token occupies it. Both the token's identity and the structural position contribute, with the latter acting independently of token type.

• RoPE is not responsible. Replacing the first token's key-side RoPE rotation index with a randomly sampled index from positions 2 through T barely changes Sink scores in any of the five models. This undercuts the intuition that the first token's positional encoding creates sinks.

• Self-concentration induces sinks at any position. Forcing the attention weights at position 16 to put all mass on itself (α = 1.0, all layers) produces both AS and MAs at position 16 across Llama-2-7b-hf, Llama-3.2-3B, Mistral-7B-v0.3, Qwen2-7B, and pythia-1b. The effect is weaker for Mistral.

• Value-non-mixing is the proposed mechanism. A uniform repeated-token sequence keeps every position in a Value-non-mixing state, and MAs appear at all positions even though attention is spread uniformly. Inserting a single distinct token breaks the state after that point, and AS/MAs retreat to the first token and the identical-token positions preceding the distinct token.

• Model-dependent strength. Three of five models (Llama-2-7b-hf, Llama-3.2-3B, Qwen2-7B) show the pattern robustly, with Sink scores of roughly 0.68–0.80 when a distinct token is present versus 0.000 for uniform sequences. Mistral-7B-v0.3 shows the same direction far more weakly (0.07–0.16), and pythia-1b shows a very small, strongly input-dependent effect (≤0.03 at every threshold tested).

• Layer-wise ordering varies by model. When the BOS token is relocated, Llama-3.2-3B shows AS and MAs from Layer 1, Mistral shows AS first then MAs, and Llama-2-7b-hf shows both phenomena only from Layers 2–3 — suggesting the causal chain differs architecturally.

• Robustness to configuration. The self-concentration intervention holds across WikiText, GSM8K, and SlimPajama, across sequence lengths of 32 to 256 tokens, across intervention positions 8/16/32, across Sink thresholds from 0.2 to 0.5, and when restricted to early layers only.

Methodology in Plain English

The authors start from a structural fact about causal language models: because a token cannot attend to the future, the very first token can only attend to itself, so its attention weight on itself is exactly 1.0. Whatever occupies position 1 gets its own Value vector passed through unchanged rather than averaged with other tokens. The authors call this "Value-non-mixing."

To test whether this is what creates sinks and massive activations, they run three kinds of experiments:

1. Swapping tokens. They move the BOS token to the middle of the sequence to see whether sinks follow the token or stay at position 1. They do both.

2. Surgical interventions. They directly edit the rotation matrix applied to the first token's Key vector to break its positional encoding, and separately force the attention distribution at position 16 to be self-only — overriding the normal softmax weights across all layers. If RoPE were the cause, the first intervention should kill the sink. It doesn't. If self-concentration were the cause, the second intervention should create a sink. It does.

3. Repeated-token sequences. Since identical tokens all have identical Value vectors, a sequence of the same token repeated T times is in a Value-non-mixing state everywhere regardless of how attention is spread. Inserting one different token at position 1, 2, or 3 creates a sharp boundary: positions before the distinct token stay "unmixed" and positions after it do not. The authors then check whether sinks and massive activations appear exactly on the unmixed side of that boundary.

The primary metric is Sink, the fraction of (layer, head) pairs whose average attention weight on a given position exceeds a threshold (default 0.3). Experiments use 100 sequences of length 64 per condition, drawn from WikiText, with GSM8K and SlimPajama used for robustness checks.

Why This Matters

This paper matters because Attention Sinks and Massive Activations are not academic curiosities — they are the specific numerical features that break low-bit quantization and complicate KV-cache compression in deployed models. If researchers understand that these phenomena arise from a structural property of causal masking combined with Value-vector composition, they may be able to design architectures or training regimes that avoid or neutralize them, rather than working around the resulting outliers after the fact.

Impact on research:

• Provides a mechanistic alternative to accounts that attribute sinks to the BOS token, key-bias behavior, or positional encodings.
• Distinguishes its "Value-non-mixing" from Barbero et al.'s "anti-overmixing," which concerns information propagation across positions and layers rather than the composition of a single attention output. The two are complementary.
• Offers a controlled causal protocol (interventions plus repeated-token inputs) that other interpretability researchers can reuse.

Real-world applications:

• Post-training quantization: Knowing which positions generate extreme activations could let practitioners apply position-aware clipping or outlier handling instead of blanket scaling.
• KV-cache compression and eviction: Sink-aware cache policies (e.g., always retaining the first token) could be replaced with policies based on the actual mechanism.
• Long-context inference: Understanding why the first token acts as an anchor informs how to handle streaming or sliding-window attention.
• Architecture design: Newer models such as Qwen3-Next and gpt-oss already emphasize reducing these phenomena; this work gives designers a concrete target.

Industry relevance: Any organization serving LLMs at scale on quantized hardware, or building efficient inference stacks, has a direct stake in controlling activation outliers.

Future Directions

• Quantifying Value-vector similarity directly. The authors leave open whether the input-dependence seen in Mistral and pythia relates to the similarity between the repeated token's Value vector and the inserted distinct token's Value vector.

• Extending Value-non-mixing beyond the sequence-initial position. The study confines itself to position 1 to isolate structural drivers. Whether naturally occurring intermediate sinks can be explained by the same framework is unresolved.

• A continuous "Value Mixing Index." The current definition is binary (mixed or not mixed). A continuous, predictive metric would need to model Value-vector geometry and prove it predicts both Sink scores and activation magnitude across models and natural inputs.

• Connecting the mechanism to downstream damage. No experiment measures whether MAs or Value-non-mixing actually affect quantization error, perplexity, or task accuracy — a gap the authors explicitly flag.

• Head-level causal pathways. The interventions operate layer-wide; pinpointing which specific heads drive the effect, and how that varies by architecture, remains open.

Target Audience

This paper is best suited for researchers and engineers working on LLM interpretability, efficient inference, and quantization. Mechanistic interpretability researchers will find the causal intervention protocol and the Value-non-mixing framework most directly useful. Practitioners building quantization pipelines or KV-cache systems will benefit from the practical implications even if they skim the formalization. Readers without a working understanding of attention mechanics, RoPE, and softmax normalization will find the methodology sections difficult, though the core intuition — that the first token only ever sees itself, so its output never blends with anything else — is accessible to anyone who understands the basics of transformer attention.

Authors’ abstract

Large Language Models (LLMs) often exhibit "Attention Sink" (AS) and the accompanying "Massive Activations" (MAs) at the initial position of a sequence. These phenomena frequently co-occur, and MAs can pose challenges for low-bit quantization. In this study, we analyze the factors underlying AS and MAs that emerge at the initial position regardless of the token occupying it. Our experiments suggest that self-concentration of attention, resulting from the causal mask, and the subsequent Value-non-mixing in attention outputs contribute to AS and MAs. These findings provide new empirical evidence on the internal dynamics of LLMs, offering insights that may inform future quantization strategies and advance our understanding of the internal mechanisms of attention layers.