A Conditional No-Hair Theorem for Transformers: How Language Models Forget Their Prompt

AI summary

Overview

Research area: Mechanistic interpretability and theory of transformer language models — specifically, how much information about a prompt survives in the residual stream as generation proceeds, and why.

Technical level: Advanced. The paper combines an empirical measurement protocol with a formal theorem (Lipschitz bounds, a dilution lemma, a layer-wise recursion, and an induction over depth). The prose is unusually accessible, but the core result is a mathematical bound.

One-sentence scope: The paper measures and proves that the hidden-state difference caused by two different prompts, when both are followed by the same continuation, decays roughly as a power law in the number of generated tokens, at a rate no slower than O(P(log t)^(L-1)/t) for prompt length P and depth L, and argues that training accelerates this forgetting.

What This Paper Is About

A transformer can attend to every token of its prompt at every step, yet in practice a system prompt or an opening instruction stops mattering after enough generation. The paper asks a precise version of this question: if you give a model two different prompts and then force both to continue along the same text, does anything inside the model still distinguish them after t tokens, and how fast does that distinction fade?

The goal is to turn an informal intuition about "instruction drift" into a measurable quantity, a decay law, and a theorem — and to identify the mechanism (a shrinking share of attention to the prompt, plus "relay" of prompt information through intermediate generated tokens) that produces it.

Key Contributions

1. A measurable definition of prompt memory. The authors define the prompt offset — the difference between hidden states of two runs that share a continuation but differ in their prompt — plus a signal and noise decomposition that yields a decodability ratio, and a set of estimators (including a bias-corrected signal estimator) for fitting power-law exponents.

2. A conditional no-hair theorem. For finite-depth pre-LN softmax transformers with full causal attention and bounded attention logits (e.g. rotary position encodings), the offset is bounded by O(P(log t)^(L-1)/t), provided the pre-norm activations stay away from zero (Assumption 1). The bound is shown to be attained by an exactly solvable linear relay model with uniform attention.

3. A path decomposition into dilution and relay. The attention contribution to the offset is split into a direct path (queries reading the prompt) and a relay path (queries reading continuation tokens that already carry prompt information). The decomposition reproduces the measured exponent within 0.06 at every layer and training stage.

4. A training-dynamics result and an application. Compensating dilution with a logit bias on the prompt removes or reverses forgetting; forgetting appears within the first billion training tokens on Pythia, before attention sinks form. A mild log t compensation raises instruction adherence at a small perplexity cost.

Main Findings

• Trained models forget faster than untrained ones. Beyond the first layers, trained models show α ≈ 0.55–0.7, while untrained models with identical architecture show α ≈ 0.2–0.5, and the untrained exponent decreases with depth. Smaller α means longer memory, so random weights retain the prompt better than trained weights.

• The decay is close to a power law in t. The offset decays approximately as t^(-α), with α = 1 being the rate at which a fixed prompt's share of uniform attention shrinks. Untrained networks show logarithmic corrections in the local exponent, which rises towards 1; trained Pythia on coherent text is close to a pure power law.

• Noise stays flat, so one exponent summarises forgetting. The spread of hidden states across different continuations has β ≈ 0 almost everywhere (the exception reported is untrained Qwen), meaning decodability decays at essentially the same rate as the signal itself.

• Dilution is necessary, and apparently sufficient. A prompt of P tokens competing against a growing context holds at most P·e^(2B)/(P+i) of a query's attention at distance i. When the authors cancel this dilution with a logit bias on the prompt, forgetting stops early in training and the offset actually grows after training. Untrained networks and the solvable linear model show dilution is also sufficient on its own.

• Relay cannot rescue the prompt. Continuation tokens passing along what earlier tokens read from the prompt adds only logarithmic factors, which is why the bound is (log t)^(L-1)/t rather than 1/t.

• Training suppresses relay early. An attention path decomposition shows that training suppresses the relay path within the first billion tokens, and that this happens before attention sinks form — so sinks cannot be the initial cause of forgetting.

• Attention weights are a misleading proxy. Sinks attract attention weight without carrying prompt-specific content, so the decay of the trace in the hidden state, not the decay of attention weights, determines whether the prompt remains readable.

• The power law is not inherited from text statistics. An untrained network, whose weights carry no corpus statistics, already forgets as a power law, and its exponents change by at most 0.01 when continuation sentences are shuffled. In trained networks the exponent does depend on the continuation text, but the ordering of the axes and the trained–untrained contrast do not.

• Compensation helps, but not because of its shape. A mild log t compensation increases instruction adherence at a small perplexity cost but induces instruction echoing; in multi-turn conversations it shows a consistent trend towards higher adherence, but no more than a constant boost with the same mean bias.

Methodology in Plain English

The central trick is to hold the text fixed. If you let two prompts generate freely, the resulting texts differ, and you cannot tell whether a difference in hidden states is memory of the prompt or just a response to different words. So the authors take two prompts — differing by one word, by paraphrase, by genre, or by language — and force both to read the same human-written continuation. Any remaining difference in the hidden states must come from the prompt. This is called teacher forcing, and the authors are explicit that it is a measurement device, not the failure mode itself.

They measure that difference at each layer, average it over a window of consecutive continuation tokens, and normalise each coordinate by its own scale so no single dimension dominates. They split the difference into a part common to all continuations (the signal) and the spread across continuations (the noise), then fit power laws on a log time axis. Measurement windows are spaced by octaves — ending at t_e = 16, 32, …, 1024 for Pythia on real-corpus pairs and 16, …, 2048 for Qwen — and labelled by their centre rather than their end, because end labelling biases the fitted exponent upwards.

Prompts come from two sources: twenty hand-written scenes with a base prompt plus four variants (detail, paraphrase, genre, language), each with sixteen human-written continuations of at least 1920 tokens from WikiText-103 validation; and real-corpus pairs, one hundred per axis, covering a one-word change, MRPC paraphrases, WMT14 English–French translation pairs, and two unrelated WikiText sentences, each with eight shared continuations.

Models are chosen to separate training, width, and depth: Pythia-160m at eight checkpoints from initialisation to the end of training (steps 0, 128, 256, 512, 1000, 4000, 16000, 143000), Pythia-410m and Pythia-1b at initialisation and after training, Qwen2.5-0.5B trained and at random initialisation, and trained Qwen2.5-1.5B. All computation runs in float32 on a single Apple-silicon laptop.

On the theory side, the proof proceeds in three steps: bound the constants the architecture imposes (bounded logits, a Lipschitz constant for LayerNorm of 2√d/r); prove that a prompt's share of attention is diluted as the number of visible keys grows; and turn this into a one-layer recursion with three terms — the offset at the same position one layer earlier, an average over all earlier positions (relay), and a direct O(P/i) prompt term. Induction over depth then gives the theorem.

Why This Matters

Impact on research. The paper reframes instruction drift and long-context degradation as a geometric consequence of a growing context rather than a quirk of particular dialogue patterns, and it makes a quantitative claim where prior work offered qualitative descriptions. It also delivers a counterintuitive result that any account of long-context behaviour has to accommodate: training makes models forget their prompt faster than random weights do, and this happens before attention sinks form. It connects three previously separate literatures — instruction drift, information flow and over-squashing, and power-law attention statistics — and offers a mechanism (dilution plus relay) that spans all three.

Real-world applications:

• System-prompt and persona stability. Quantifying how long a persona set at the start of a context remains readable tells practitioners how often to restate or re-inject instructions.
• Long-running agents. An agent whose constraints are given once at the start of a long trajectory undergoes the same erosion; knowing the decay law informs whether constraints need periodic re-anchoring.
• Context-length and attention-scaling design. The analysis predicts that logit scalings which grow with context length, such as Scalable-Softmax, fall outside the theorem's scope because the logits become unbounded, and should therefore retain prompt information longer — a concrete design hint.
• Evaluation and monitoring. The promised toolkit, which measures α for any Hugging Face causal language model, turns prompt retention into a benchmarkable property rather than a matter of anecdote.

Industry relevance. The results bear directly on system-prompt engineering, multi-turn product design, and the choice of attention-scaling schemes for long-context models. The finding that attention weights are an unreliable proxy for information flow also warns against optimising attention visualisations or sink behaviour as if they measured memory.

Future Directions

• Extending the theorem beyond its scope. The proof covers pre-LN decoders with full causal attention and bounded logits; it explicitly excludes post-LN blocks, linear attention, and recurrent state. Whether the 1/t rate survives in those architectures is open.
• Relaxing Assumption 1. The bound requires pre-norm activations to stay away from zero. The empirical margins reported are comfortable at block inputs (for example r/√d = 0.11 for Pythia-160m at initialisation and 0.49 for trained Pythia-1b), but the Qwen2.5-0.5B constant is 0.012, set by the first post-attention norm. The authors note their checks cover only the measured sequences and lengths.
• Testing the prediction about context-growing logit scalings. If unbounded logits break Lemma 1(a) and with it Lemma 2, then Scalable-Softmax and similar schemes should retain prompt information longer. This is a testable claim the paper states but does not, in the content summarised here, measure.
• From teacher forcing to free generation. The measurement device fixes the continuation; the actual failure mode is free generation, where the prompt's influence can only travel through the text the model itself has caused. The paper points to its Section 8 for this setting, leaving the relationship between the measured exponent and free-generation drift as a question for further work.

Target Audience

Researchers working on mechanistic interpretability, long-context modelling, and the theory of transformer computation will get the most from this paper, particularly those interested in formal guarantees about information propagation. Practitioners who design system prompts, long-running agents, or multi-turn dialogue products will find the empirical results — especially the trained-versus-untrained contrast and the unreliability of attention weights as a proxy — directly usable even without following the proofs. Readers need comfort with power laws, Lipschitz arguments, and attention notation to engage with the theorem itself; the introduction and the measurement sections are readable without them.

Authors’ abstract

We ask how much an autoregressive transformer retains of its prompt after generating $t$ tokens. Consider two different prompts followed by the same continuation text, so that any difference in their hidden states comes from the prompt alone, and we find that the distance between their hidden states decays approximately as a power law $t^{-α}$, where $α=1$ is the rate at which a fixed prompt's share of uniform attention shrinks and smaller values mean longer memory. Trained models forget fast ($α\approx0.55$--$0.7$), untrained ones slowly ($α\approx0.2$--$0.5$). We prove that this offset is $O\big(P(\log t)^{L-1}/t\big)$ for a prompt of length $P$ and depth $L$. The bound holds for finite-depth pre-LN softmax transformers with bounded attention logits, under one additional condition: the pre-norm activations stay away from zero. The mechanism is geometric: the prompt's direct influence is diluted as the context grows, and \emph{relay} --- continuation tokens passing on what earlier tokens read from the prompt --- can only add logarithmic factors. This is what we call the \emph{conditional no-hair theorem}: given the text generated so far, the prompt's influence on the next token vanishes as $t$ grows, so any property of the prompt that survives, such as its language, must be carried by the text itself. An exactly solvable linear relay model shows that the bound cannot be improved and reproduces the rise of the local exponent towards $1$ seen in untrained networks. Cancelling the dilution with a logit bias stops the decay early in training and reverses it after, so dilution is necessary for the forgetting we measure; untrained networks and the solvable model show it is also sufficient.