Self-Generated Feedback Destabilizes Test-Time Training: A Causal Decomposition of Long-Horizon Adaptation

AI summary

Overview

• Research area: Natural Language Processing — test-time training (TTT), long-context language models, and inference-time learning.
• Technical level: Advanced (assumes familiarity with language-model training, gradient updates, fast weights, and long-context evaluation).
• Scope: The paper runs matched long-horizon experiments (128K-token streams) on TTT-E2E models at 125M, 760M, and 3B scale plus Adam updates on Qwen3-4B, and decomposes why learning from a model's own generated text degrades prediction on independent human-written text.

What This Paper Is About

Test-time training lets a language model keep updating part of its weights during inference, but when the model also generates the text it later trains on, each update changes the model that produces the next training example. The authors ask which part of this closed loop causes harm, using controlled comparisons that separately vary the generator, the text being read, and whether updates are retained. They then propose and test a rule that evaluates a candidate update on independent real text before committing to it.

Key Contributions

1. Long-horizon evidence of damage from self-generated writes: Across three TTT-E2E configurations (125M, 760M, 3B) and under ordinary Adam on Qwen3-4B's existing feed-forward weights, retaining generated-text updates worsens prediction on independent human-written text, while the same update rules improve prediction when trained on real text.
2. A causal decomposition of the feedback path: Three matched comparisons — Fixed Generation, Recorded Replay, and paired one-update branches — separate changes to future training text, harm from reading degraded text in attention, and additional harm stored in the weights.
3. A transfer criterion called Settlement: A candidate state is compared against the current state on independent real text before the update is committed, which leaves near-zero endpoint gaps while retaining real-text adaptation.
4. Characterization of where damage concentrates: Most large late-stream failures come from a small subset of source sequences rather than from all late passages, and gradient conflict between generated-chunk and real-passage gradients ranks which single updates are more harmful.

Main Findings

• Closed-loop writes worsen real-text prediction across scales: Retaining generated-text updates adds +3.01 nats [ +1.86, +4.20 ] at 125M, +6.00 [ +4.27, +7.73 ] at 760M, and +0.40 [ +0.30, +0.51 ] at 3B over Writes Off, using the same six books, five seeds, and 128K horizon at every scale. Every interval excludes zero. The authors describe this as a cross-scale replication of direction, not a monotone relation with parameter count.
• The damage is cumulative, not immediate: Within the earlier 8K+8K evaluation window, Closed Loop and Writes Off remain close; the gap grows only after many additional self-writes.
• Decoding changes slow but do not fix it: Full-support sampling reduces the 128K endpoint gap from 3.010 to 1.452 [ .708, 2.180 ] but late loss still rises. An eight-chunk reset of the fast weights leaves a .175 gap [ .084, .321 ] but preserves only 44.1% of the benefit of real-text adaptation.
• Real text interrupts the loop: Replacing 0, 5, 11, 21, or 33 of the stream's 105 generated chunks with real chunks reduces harm to approximately 87%, 37%, 7%, and 6% respectively. Grouping 33 chunks into bursts instead of spacing them evenly leaves about 60% of the original harm rather than 6%.
• The failure is not specific to TTT-E2E: With Adam on Qwen3-4B feed-forward weights at learning rate 10^-4, persistent self-writing raises real-text NLL by 1.231 nats [ .580, 2.350 ], while the same 10^-4 update lowers NLL by .168 [ .132, .204 ] on real text. At 10^-5 the interval includes zero ( .026 [ -.071, .122 ] ).
• Writing can still help: On eleven identical teacher-forced real-text streams, retaining updates lowers mean NLL by .0340 nats relative to No Updates.
• Fixed Generation removes nearly all the damage: With a frozen W_0 generating every chunk for a separate learner, Fixed Generation leaves .052 nats at 125M and .073 nats at 760M of excess change, removing 98.3% and 98.8% of the matched Closed Loop gaps. Update magnitude does not explain the result: Fixed Generation moves farther from the initial weights ( .01387 ) than Closed Loop ( .00857 ), while Real-Text Learning moves farther still ( .01759 ) and improves NLL by .3544 nats [ .1236, .6048 ].
• Both reading and writing hurt when text is held fixed: In Recorded Replay at 125M, reading a saved degraded stream without updates adds 1.606 nats, reading and writing adds 3.874 nats, and the extra 2.268 nats is the cost stored by the weight updates. A 3B replication measures a write-minus-read cost of .1927 nats [ .1043, .3272 ], positive for every source- and receiver-book aggregate.
• One update can fit its source and harm new text: In paired one-update branches at 125M, keeping versus discarding an update lowers training-chunk NLL ( −.1374 after Writes Off history; −.0354 after Closed Loop; −.1449 after Fixed Generation) while raising next-real-text NLL ( +.0045, +.0381, +.0018 respectively). Within Closed Loop, the mean one-update cost grows from 0.006 nats at the earliest measured position to 0.113 at the latest.
• Gradient conflict ranks harmful updates: Across 96 paired updates per model-history setting, the cosine between generated-chunk and real-passage gradients correlates with measured harm at −.502 Closed Loop / −.788 Writes Off at 125M and −.633 / −.731 at 760M; the first-order predicted loss change correlates at +.704 / +.794 at 125M and +.632 / +.751 at 760M. The authors note this uses the real passage on which harm is measured and is not a selection rule in these experiments.
• Damage concentrates in a few trajectories: Among 72 paired costs at each position, the first position has mean 0.0439 and median 0.0422 with no value above 0.5 nats; the final position has mean 0.7512 but median 0.0064. Only 12 of 72 late costs exceed 0.5 nats, and all twelve come from four of 24 source sequences. Once a source–receiver pair exceeds the threshold, later sampled passages for that pair remain above it.
• Milder controls leave substantial damage: Repetition weighting and read-only real context leave 54% and 48% of the Closed Loop gap, while an eight-chunk reset removes 94.2% of the canonical gap but retains only 44.1% of useful adaptation.
• Settlement nearly closes the gap: Settlement accepts 22/936 candidates at 125M and 18/312 at 760M and leaves endpoint gaps of .07 nats at 125M and −.02 nats at 760M, with both intervals including zero. Without validation, quarter-sized writes, read-only real text, and their combination leave .90, 1.04, and .28 nats; adding Settlement puts every variant within .04 nats of Writes Off.
• Settlement retains real-text learning: On a matched all-real stream it accepts 89 of 113 candidates, reduces the first-to-last NLL change by .0534 nats [ .0299, .0721 ] relative to Reject All, and has a paired advantage of .0083 [ .0020, .0161 ] over Write All.
• Validation beats label masking under noisy labels: With correct source labels, Source Masking is slightly lower in NLL ( 3.8072 versus 3.8196 ). When labels are corrupted, Source Masking's NLL rises, while Settlement's mean NLL is lower from 21.2% observed error and paired intervals exclude zero at evaluated error rates of 38.9% and above.
• Task-metric validation helps agents: On WebShop, Settlement accepts 10 of 180 candidate blocks and reaches .1853 exact success versus .1053 for Closed Loop and .1600 for the shared Writes Off reference. Breaking feedback raises mean exact success from .1053 to .1800 across five paired seeds. On ALFWorld, two of three Closed Loop seeds complete none of the 134 unseen tasks, while Settlement reaches 88.1%–94.0% unseen-task success across seeds, retaining 19%–33% of update blocks across the two scales.
• Repetition penalty alone is insufficient: A repetition penalty reduces the 128K endpoint gap from 3.01 to .88 nats but does not identify which passage is safe to write.

Methodology in Plain English

The authors build a controlled comparison between two policies that see the same things except for one difference. Under "Closed Loop," the model generates a chunk of text, learns from it, and keeps the resulting weight update, so those new weights help generate the next chunk. Under "Writes Off," the model reads the same kind of generated chunks but throws the update away. Because both start from identical weights and an identical real-text prefix, the difference between them measures what persistence of generated-text writes causes.

Each 128K-token stream starts with 8K tokens of real text, then the model generates 105 chunks of 1024 tokens (written as K). The stream is paused 15 times plus an initial reference to evaluate the current weights on independent human-written passages, with the preceding weights and attention state restored afterward so evaluation cannot alter the stream. Generation uses temperature one and top-p = 0.95 nucleus sampling. The main cross-scale comparison uses the same six PG-19 books, five seeds, 128K horizon, and batch width eight at 125M, 760M, and 3B.

To find the cause, they run three matched interventions. Fixed Generation keeps the learner updating but has a frozen initial model generate all future chunks, breaking the link from updates to later training text. Recorded Replay saves a degraded stream and has a fresh receiver process the identical tokens first read-only and then with updates retained, holding content fixed. Paired one-update branches copy the same weights and attention cache just before a chunk so that the only difference is whether that single update is kept.

Finally, they define Settlement: given current fast weights W, it forms a temporary candidate W + δ, compares both on the same independent real-text passage without changing attention, and commits δ only when the candidate's loss is no better-and-worse (A ≥ 0). Candidates are evaluated on top of earlier accepted updates. The rule does not use the candidate's source label but requires identifiable real text for comparison. Figures and tables report 95% intervals that resample books after averaging seeds within each book.

Why This Matters

• Impact on research: The paper reframes a common assumption — that lower loss on the text an update was trained on demonstrates the update is useful. It provides a causal decomposition showing that the distinguishing feature of the failure is that updated weights determine later training text, and it shows that generated origin alone is insufficient, since Fixed Generation learns from synthetic text without damaging real-text prediction.
• Real-world applications:
  • Long-document and streaming assistants that would otherwise adapt on their own outputs over extended sessions.
  • Long-running agents that learn from their own actions, where every candidate carries the same "generated" label and label masking cannot distinguish them.
  • Systems needing graceful behavior when source labels are noisy or unavailable, where state-based validation is more robust than label filtering.
  • Deployment settings where retaining useful adaptation matters, since Settlement preserves measurable real-text learning rather than disabling inference-time updates.
• Industry relevance: The results apply beyond TTT-E2E — the same failure appears with Adam on Qwen3-4B's existing weights — so they are relevant to any production system that persists inference-time updates from self-generated text. The cost trade-off is explicit: milder controls leave substantial damage (54% and 48% of the gap), while an eight-chunk reset removes 94.2% of the gap but keeps only 44.1% of the adaptation benefit. The project repository is publicly available at https://github.com/lingjivoo/ttt-ouroboros.

Future Directions

1. Make the conflict signal usable as a selection rule: The gradient-cosine and first-order predicted-loss scores rank harm but currently use the real passage on which harm is measured; a version that does not require that passage is not reported.
2. Reduce the dependence on available real text: Settlement requires identifiable independent real text (or retained prefix text) at commit time; behavior when no real text is available is described only through the fallback to prefix text.
3. Address the state-dependent failure directly: Because 12 of 72 late high costs come from four of 24 source sequences that stay harmful once they cross the threshold, detecting and handling persistently bad trajectories is an open problem the paper does not resolve.
4. Separate state change from text change in the growing one-update cost: The authors note that within Closed Loop both the model state and its generated text change, so the comparison measures their combined effect rather than isolating either one.

Target Audience

Researchers and engineers working on long-context language models, test-time training, fast weights, continual adaptation, and agents that learn during deployment. It is also relevant to practitioners deciding whether to persist inference-time updates and to those studying generated-data feedback and model collapse, since the paper distinguishes its within-stream setting from recursive synthetic-data training across generations. The controlled design and interval reporting make it useful for readers who want an experimental template for separating feedback, attention, and weight effects.

Authors’ abstract

Test-time training (TTT) lets a model store information in its weights during inference. When the model learns from its own output, however, each update also changes the model that generates the next training example. Across 128K-token streams, retaining generated-text updates worsens prediction on independent human-written text with three TTT-E2E model configurations (labeled 125M, 760M, and 3B). The same failure occurs when Adam updates Qwen3-4B's existing weights. The same update mechanisms can improve on real text, so writing itself is not the failure. Three matched comparisons trace the causal pathway. Fixed Generation removes over 98% of the damage at 125M and 760M by using a frozen model to generate training chunks. Recorded Replay separates the loss caused by reading degraded text from the additional loss stored by updating on it. A paired one-update comparison then shows the local conflict: an update predicts its source better but new real text worse. This cost grows after Closed Loop adaptation, with a few trajectories accounting for most large failures. Finally, Settlement evaluates the candidate state on independent real text before commitment. It leaves mean endpoint gaps of 0.07 and -0.02 nats at 125M and 760M while retaining real-text adaptation. These results motivate checking prediction on independent evidence before retaining an update.