Position, Masking, and Sequence Geometry

Feed a self-attention layer the same set of vectors in a different order and its outputs come back permuted the same way. Position or structure has to be injected; the mechanism compares content, not chronology. That is not a rule of thumb about attention. It is a published property, with a formal definition and a named proposition behind it.

The Set Transformer paper wrote both down in 2019. Its Definition 1 is exact: a function f : X^n -> Y^n is permutation equivariant iff f(pi x) = pi f(x) for every permutation pi in S_n. The paper then applies that definition to the two positionless self-attention operations it builds, the Set Attention Block (SAB) and the Induced Set Attention Block (ISAB). The sentence immediately before Property 1: “We also emphasize that both of our set operations (SAB and ISAB) are permutation equivariant”. Shuffle the encoder's input and its output arrives shuffled in exactly the same way. Nothing else about it changes. Nothing else in it knows the order changed.

A second group, working on an unrelated problem, states the same fact in its own words. The T5 paper, 2020: “Since self-attention is order-independent (i.e. it is an operation on sets), it is common to provide an explicit position signal to the Transformer.”

So the architecture has to supply the geometry — sequence order, spatial coordinates, graph distance, whatever the task runs on. Equivariance is the default. The design has to overturn it deliberately.

Each scheme chooses different assumptions about distance, length, and extrapolation. Four of the rows below come with published measurements. Three come with a verdict from the authors who ran the experiment.

The original Transformer chose sinusoids in 2017, for a stated reason: “we hypothesized it would allow the model to easily learn to attend by relative positions”. The same paragraph records the control they ran. They “also experimented with using learned positional embeddings instead, and found that the two versions produced nearly identical results”. So the measurement did not decide it. The sinusoidal form survived on a second argument instead — “because it may allow the model to extrapolate to sequence lengths longer than the ones encountered during training”. That is a hypothesis about a regime they had not measured.

A 2018 follow-up measured what the relative row is worth. In the big configuration, relative position representations gained 1.3 BLEU on WMT 2014 English-to-German over absolute ones, and 0.3 BLEU on English-to-French. Base and big gains were 0.3 and 1.3 BLEU for EN-DE, 0.5 and 0.3 BLEU for EN-FR. The sharper number is the clipping-distance ablation, scored on the English-to-German development set newstest2013. Remove relative position information entirely, at clipping distance k = 0, and BLEU collapses to 12.5. Restore the crudest possible relation, k = 1, and it returns to 25.5. Every larger clip lands between 25.8 and 25.9 (k = 2: 25.8, k = 4: 25.9, k = 16: 25.8, k = 64: 25.9, k = 256: 25.8). The base model used k = 16, the big model k = 8. Almost all of the value is in knowing that a neighbour is a neighbour. Stacking a second scheme on top buys nothing: “Notably, we observe that combining relative and absolute position representations yields no further improvement in translation quality.”

The relative row also has production numbers. T5 dropped absolute position embeddings for a simplified relative scheme. Each "embedding" is a single scalar added to the attention logit, shared across all layers but distinct per head. The 2020 paper states the sizing: “In this work, we use 32 embeddings for all of our models with ranges that increase in size logarithmically up to an offset of 128 beyond which we assign all relative positions to the same embedding.” The consequence is structural. Past an offset of 128 tokens, a single layer cannot tell one distance from another. Only stacked layers recover longer offsets.

Rotary encoding came later. RoPE encodes “the absolute position with a rotation matrix”, while incorporating “the explicit relative position dependency in self-attention formulation”. It was posted in 2021 and later published in Neurocomputing.

The warning in the coordinate row — that a scheme may expose unwanted absolute cues — has been measured too. Swin Transformer, in 2021, was built on a learned relative position bias over the (2M-1) x (2M-1) grid of within-window offsets. Its ablation: “Swin-T with relative position bias yields +1.2%/+0.8% top-1 accuracy on ImageNet-1K, +1.3/+1.5 box AP and +1.1/+1.3 mask AP on COCO, and +2.3/+2.9 mIoU on ADE20K in relation to those without position encoding and with absolute position embedding, respectively, indicating the effectiveness of the relative position bias.” On the same backbone, absolute position embedding helps ImageNet classification by +0.4% top-1 while hurting localisation: -0.2 box and mask AP on COCO, -0.6 mIoU on ADE20K. One added cue, two opposite signs. Which sign you get is decided by which invariances the task has.

Several masks may be combined before normalization, and the combination can be the architecture. Big Bird is made of exactly three composable masks: g global tokens attending to the whole sequence, a sliding window of w local neighbours, and r random tokens. Eleven authors at Google Research published it in 2020. That sum is not a heuristic. They report it is a universal approximator of sequence functions and Turing complete, and they run it at sequence length 4096. What the sparsity buys is in the abstract: “The proposed sparse attention can handle sequences of length up to 8x of what was previously possible using similar hardware.”

Each layer below is a set of permitted edges. A real system is their intersection.

Many mask failures are silent because tensor shapes remain valid. The first of them is a deliberate design decision at a frontier lab. Llama 3 masks attention across document boundaries inside a packed sequence, and Meta's July 2024 report says why: “We use an attention mask that prevents self-attention between different documents within the same sequence. We find that this change had limited impact during in standard pre-training, but find it to be important in continued pre-training on very long sequences.” The typo is in the original.

The cost of omitting that mask has been measured independently. An ACL 2024 study found that plain causal masking over concatenated documents admits distracting information from previous documents. Its related-document packing method, BM25Chunk, improves in-context learning by 11.6%, knowledge memorisation by 9.8% and context utilisation by 7.2%. None of that shows up as a shape error.

A small synthetic batch can reveal most access-control errors. The edge sets worth asserting are the ones the published architectures name explicitly: document boundaries inside a packed sequence; the g global tokens, the w-wide local window and the r random tokens of a sparse pattern; the sub-window partition a cyclic shift creates. Each of those is an adjacency matrix. You can write it down before training and compare against it.

Absolute time can matter in seasonality. Relative intervals matter in event dynamics. Two-dimensional images need both axes. Graphs may require shortest-path, edge, or spectral information.

The strongest evidence that this is a design decision, rather than a component to install, is a controlled comparison in which the brand names lose to nothing at all. A 2023 study trained roughly 107M-weight decoder-only Transformers from scratch on ten tasks in three groups. Two were primitive: copy and reverse. Six were mathematical and reasoning tasks: addition, polynomial evaluation, sorting, summation, parity and LEGO. Two were the classical length-generalization datasets, SCAN and PCFG. Five position treatments ran across all ten: sinusoidal absolute position embedding (APE), T5's relative bias, ALiBi, rotary, and no positional encoding at all (NoPE). The learnable APE variant was left out deliberately, “as the learnable variant cannot produce embeddings for unseen positions”.

On the aggregate mean-reciprocal-rank across the ten tasks, NoPE ranks first at 0.69. T5's relative bias follows at 0.55, ALiBi at 0.50, rotary at 0.33, APE at 0.22. NoPE also requires no additional computation. The authors' own summary, at NeurIPS 2023: “Our findings reveal that the most commonly used positional encoding methods, such as ALiBi, Rotary, and APE, are not well suited for length generalization in downstream tasks.”

Do not import a text positional scheme merely because the backbone is a Transformer. State which transformations should preserve behavior and which should change it. Then check that the scheme you picked was measured under those conditions, and not others.