Attention: Content-Based Interaction

For each receiving position, attention asks which source positions are relevant and what information should be mixed from them. Relevance is computed from learned query–key interactions.

The values carry the content being combined. Keys influence selection, while values determine what the weighted mixture contains.

The question was first posed in this form in 2014, by Bahdanau and two colleagues, in a paper that went on to ICLR 2015. Their introduction states the mechanism in a single sentence. “Each time the proposed model generates a word in a translation, it (soft-)searches for a set of positions in a source sentence where the most relevant information is concentrated.”

The size of the answer is in their Table 1, on WMT'14 English-to-French. RNNsearch-50 scores 26.75 BLEU on all test sentences, and 34.16 excluding unknown words. RNNencdec-50 is the same size, but forced to route the whole source sentence through one fixed vector. It scores 17.82 and 26.71. The phrase-based Moses baseline sits at 33.30, which RNNsearch-50 passes on the unknown-word-free measure and not on the full test set. A Stanford group measured the effect independently the following year: “a significant gain of 5.0 BLEU points over non-attentional systems”, and 25.9 BLEU on WMT'15 English-to-German. Two groups, two language pairs, the same direction of effect.

One attention head transforms an input sequence into context-dependent outputs: project Q, K and V; compute query–key dot products for the allowed pairs; scale and mask; normalize with softmax across candidate source positions; then form one weighted value sum per query position.

The order is not stylistic. The PyTorch 2.14 documentation prints reference code for torch.nn.functional.scaled_dot_product_attention, under the heading “Efficient implementation equivalent to the following”. It computes query @ key.transpose(-2, -1) * scale_factor. It adds the mask tensor attn_bias to that product. It applies softmax to the sum. Only then does it multiply by value.

The names describe how vectors participate in attention, not fixed semantic content. A query represents what the receiving position is seeking, one per destination position, compared against candidate keys, and it determines a row of attention scores. A key represents how a source position advertises relevance, one per source position, shaped by positional context, and it does not carry the final mixed content alone. A value represents the information a source contributes when selected: weighted by the normalized scores, combined into the output, free to have a different dimension, and able to encode contextual content.

That freedom of dimension is a real degree of freedom even where a given model declines to use it. “Attention Is All You Need” sets “dk = dv = dmodel/h = 64” across its “h = 8 parallel attention layers, or heads”. The two dimensions coincide there by construction, not by necessity.

A large negative score is added before softmax, and forbidden pairs then receive near-zero weight. This is not a rule of thumb that implementations approximate. It is the literal published operator semantics, in two specifications maintained independently of each other.

PyTorch's reference implementation builds a tensor called attn_bias. It fills the masked entries with float("-inf") via masked_fill_, adds the tensor to query @ key.transpose(-2, -1) * scale_factor, and applies softmax to the result. When scale is None, the scale_factor is 1 / math.sqrt(query.size(-1)). Setting is_causal produces a lower-triangular (tril) mask.

ONNX's Attention operator, introduced at opset 23 and revised at 24 and 25, arrives at the same defaults through a different codebase. The scale default is written into the spec: “Scaling factor applied to Q*K^T. Default value is 1/sqrt(head_size)”. Its attn_mask input accepts either shape of mask — “A boolean mask where a value of True indicates that the element should take part in attention or a float mask of the same type as query, key, value that is added to the attention score.” And is_causal=1 is defined there as bottom-right, offset-aware masking, in which “query i attends key j iff j <= i + offset”. Two projects, no shared implementation, one contract.

As vector dimension grows, unscaled dot products can have larger variance. Very large score magnitudes push softmax toward sharp, saturated distributions with weak gradients.

Scaling moderates this effect under common initialization assumptions. It does not guarantee balanced attention or prevent all numerical issues.

The factor comes from a variance calculation, stated in a footnote. Vaswani and seven co-authors published it in “Attention Is All You Need” in 2017. “To illustrate why the dot products get large, assume that the components of q and k are independent random variables with mean 0 and variance 1. Then their dot product … has mean 0 and variance dk.” The same paper sets out the model it was measured on. There are eight parallel heads at dk = dv = dmodel/h = 64. A big configuration reaches 28.4 BLEU on WMT 2014 English-to-German. That took 3.5 days on eight NVIDIA P100 GPUs.

What saturation costs, when the moderation is not enough, has since been measured at scale. Google's 2023 paper on scaling vision transformers to 22 billion parameters reports that models of about 8 billion parameters diverged after a few thousand training steps. The diagnosis: “It was caused by extremely large values in attention logits, which lead to (almost one-hot) attention weights with near-zero entropy.” An appendix puts a number on it — “without normalization, attention logits quickly grow to over 50000 in magnitude, resulting in one-hot attention weights after the softmax, and subsequently unstable training losses and gradients” — and the repair was to apply LayerNorm to queries and keys before the dot product. Zhai and colleagues at Apple hit the same failure independently, named it entropy collapse, and proved that the lower bound on attention entropy “decreases exponentially fast with the spectral norm of the attention logits”. A saturated softmax is not a stylistic complaint about sharp attention maps. It is a training run that stops.

Multi-head attention projects inputs into several lower-dimensional query, key, and value spaces, and each head computes its own mixture before outputs are concatenated and projected.

Heads may specialize, overlap, or become redundant. A head index does not carry a stable universal meaning across models or training runs.

The redundancy has been tested, not assumed. In 2019 Michel and two colleagues published Are Sixteen Heads Really Better than One? Their answer: “in practice, a large proportion of attention heads can be removed at test time without significantly impacting performance, and that some layers can even be reduced to a single head”. Machine-translation models split the difference by layer type. In them “the self-attention layers can be significantly pruned, while the encoder-decoder layers are more dependent on multi-headedness”.

The same redundancy shows up under a different instrument. Voita and four colleagues, also in 2019, pruned encoder heads with stochastic gates and a differentiable relaxation of the L0 penalty. Their abstract gives the headline number: “For example, on the English-Russian WMT dataset, pruning 38 out of 48 encoder heads results in a drop of only 0.15 BLEU.” On OpenSubtitles the margin is wider still — “we lose only 0.25 BLEU when we prune all but 4 heads out of 48”. The heads that survived were not arbitrary. They fell into three named roles: positional, syntactic, and attention to rare words. A positional head is one whose maximum attention weight lands on a specific relative position, in practice -1 or +1, at least 90% of the time. A different group repeats the finding in the survey A Primer in BERTology: “Voita et al. (2019b) showed that all but a few Transformer heads could be pruned without significant losses in performance.”

So the caution is not that heads never mean anything. Some plainly do. Clark and three colleagues scored individual BERT-base heads against gold dependency labels and reported what they found in the abstract. “For example, we find heads that attend to the direct objects of verbs, determiners of nouns, objects of prepositions, and coreferent mentions with remarkably high accuracy.” Head 8-11 reaches 94.3% accuracy on the det relation, against a 51.7 baseline. Head 8-10 reaches 86.8% on dobj against 40.0. Head 9-6 reaches 76.3% on pobj against 34.6. Head 7-6 reaches 80.5% on poss against 47.7. Head 4-10 reaches 82.5% on auxpass against 40.5. On CoNLL-2012 antecedent selection, head 5-4 reaches 65% overall, against 52% for a head-word-match baseline and 69% for the rule-based system. Which is why A Primer in BERTology records that “Clark et al. (2019) identify a BERT head that can be directly used as a classifier to perform coreference resolution on par with a rule-based system”.

Both results are true of the same architecture. Particular heads in particular trained models carry roles specific enough to be named and scored. Most heads in a trained model can be deleted for a fraction of a BLEU point. Neither licenses reading a head index as a concept in a model nobody has measured.

Attention alone is not a complete transformer block. Transformers combine attention with feed-forward networks, residual paths, normalization, position information, and repeated layers.

Position information is the ingredient this lesson has left unspecified, and it is unavoidable. Query–key scores depend only on content, so order has to be supplied separately. The scheme now standard arrived in April 2021 as Rotary Position Embedding, in the RoFormer paper. Its abstract: “Specifically, the proposed RoPE encodes the absolute position with a rotation matrix and meanwhile incorporates the explicit relative position dependency in self-attention formulation.” Position enters as a rotation, so the dot product itself carries relative distance.

EleutherAI reimplemented it the same month and reported its own measurements rather than the paper's: roughly 30% faster convergence than learned absolute position embeddings on billion-parameter models, and a 10–20% improvement over T5 relative position encoding. On cost, they wrote: “we find that rotary embeddings impose a 1-3% overhead across a range of transformer sizes”.

Path 6 will examine these architectures in depth. Here, the essential primitive is content-dependent weighted interaction between positions.