Research
Language Models Can Rewrite Their Own Weights
A deployed language model is normally frozen: making it better requires another training run, and anything a user changes by prompting disappears when the task changes.
- arXiv
- 2611.21885
- Published
- 2026-11-16
- Authors
- Jiajun Fan, Neil He, Changyi Yang, Andy Mao, Xi Zhu, Chaoran Cheng, Kaiqu Liang, Shuaike Shen, Zetian Li, Ruihan Guo, Ge Liu
AI summary
Overview
Research area: Language model self-improvement, model editing, and weight-level adaptation without gradient-based training.
Technical level: Advanced. The paper assumes familiarity with transformer internals (attention and MLP output matrices, hidden states, residual streams), linear attention, and statistical testing on paired samples.
Scope: The paper introduces and evaluates "self-rewriting," a setting in which a language model inspects, tests, and overwrites its own weight tensors through reasoning and code execution alone — no gradients, no optimizer, no training pipeline — and shows that this ability to improve and repair itself appears only at the largest tested scale.
What This Paper Is About
A deployed language model is normally frozen: making it better requires another training run, and anything a user changes by prompting disappears when the task changes. The authors ask whether a model can instead edit its own weights directly — diagnosing why it fails, writing new values into its own tensors, and testing the result against its unedited self — using only forward passes and its own reasoning. They demonstrate that a 27B open model can do this to itself, both to improve its accuracy on unseen problems and to repair a copy of itself whose weights have been corrupted.
Key Contributions
-
A new paradigm: self-rewriting. A formal setting in which a model is given its own weights, its graded answers, and the ability to run itself, while gradients, optimizers, GPUs, network access, and model loading are explicitly blocked — so every value changed is written by the model's own reasoning (Section 2). The objective is defined as producing an edit whose accuracy on an unseen test distribution exceeds the base model's.
-
A model that rewrites itself for the better. The 27B model's own edit makes it 6.6 points more accurate on 490 held-out problems with eight answers each, in a paired test fixed in advance. A second session's edit yields 4.0 points; both are significant at p < 10⁻⁵. Over the first eight sessions of the second protocol, the model withdraws in five and submits in three, and the accuracy of the model in use only rises.
-
A model that repairs a damaged copy of itself without training. Told only that some values in a copy of its weights were corrupted — not which, where, or how — the 27B model locates the fault and writes replacement values that are uncorrelated with the originals, never reading its undamaged weights. Found and tested on text math problems only, the repair carries over to six image benchmarks the session never saw.
-
Evidence that self-rewriting emerges with scale. Across one model family at 2B, 4B, 9B, and 27B parameters under the same task and tools, the 2B model makes no edit, the 4B and 9B models perturb themselves at random and break themselves, and only the 27B model improves and repairs itself.
Main Findings
-
A significant gain on unseen problems. On 490 held-out problems with eight answers each, the edit from session 8 raises accuracy by 6.6 points and the edit from session 2 by 4.0 points, each at p < 10⁻⁵; the sessions reach 61.3% and 63.8% respectively. In the first protocol, the 27B model's edit is significant on both its own task (22.4% to 29.6%) and held-out problems (57.2% to 58.3%).
-
The larger gain comes from a single uniform change. The 6.6-point edit raises the time-step bias of every linear-attention layer by 0.5, which reduces how often answers run out of tokens at the 16,384-token limit.
-
The self-edit is small and rank-one. The first-protocol edit removes 5% of a measured direction from the two output matrices of layer 8 (W ← W − 0.05·uuᵀW), a change of 0.6% and 0.1% of their norm. The direction is the difference between the average layer-8 hidden state of 20 answers that finish and 20 that run out of tokens.
-
Repair restores a broken copy. Told only that values were corrupted, the 27B model takes its damaged copy from 0.0% to 33.0% held-out accuracy, against 58.5% for the base model. Its change to the single matrix it named reaches 55.0%; its changes to the other 19 matrices alone leave the copy broken — evidence the model identified the actual cause.
-
The repair transfers to tasks never used to find it. On six image benchmarks (100 problems each, pooled), the repair raises accuracy from 7.3% to 70.2%, 9.6 times the damaged copy's accuracy (p < 10⁻⁹⁹), with p < 10⁻¹² on each benchmark. On ScreenSpot-v2 it goes from 0% to 89%, against 93% for the base model — roughly 83% of the base model's accuracy.
-
The model measures rather than guesses. At the same step, the 4B and 9B models add random noise scaled by 0.01 to their weights and, when a test shows the noise hurts, guess at another change. The 27B model reads its own hidden states, edits along a direction it measured, and reasons about where to apply it when a test fails.
-
Scale separates success from self-harm. The 4B model withdrew its edits; in its own session's tests, the edited model solved none of the problems while the base model solved 40–50%. The 2B model made no edit at all.
-
Cheaper tests produce better edits. At the full 16,384-token budget, tests are slow, the model runs few of them, estimates its direction from a single pair of answers, and withdraws an edit that misses the task's bar. At 4,096 tokens it runs many more tests and submits an edit that is significantly better in its own tests.
Methodology in Plain English
The researchers give the model a "session" on its own weights. The model receives a starting diagnosis: it answers some practice problems with its own weights and is told how it fails — its accuracy, how many answers run out of tokens or end in repetition, and how often its finished answers are right.
It then takes one action per turn. It can run Python code over the weight tensors (read and replace values), generate answers to practice problems with the unedited weights, the edited weights, or both, and read its own hidden states on text of its choice. A helper function computes W − α·uuᵀW, which weakens by a fraction α whatever a matrix writes along a chosen direction of the hidden state.
Crucially, the environment forbids training. Gradients are switched off and any call computing them or running an optimizer is refused; the session has no GPU, cannot load a model, start a process, or use the network; edits to the input embedding, the last normalization, and the output layer are refused, as are edits that merely rescale a whole tensor (which would act like a temperature change). A session gets 20 forward passes and 200 code runs, and the model is not told the turn limit.
When a "generate" action runs both the edited and unedited weights on the same problems with the same seeds, the answers form pairs, and the model gets a paired statistic: the mean per-problem difference in accuracy, its standard error, the change in answers that run out of tokens, and counts of answers that turn wrong-to-right and right-to-wrong. The model decides whether to submit an edit or leave none — leaving none is how it withdraws. In a second protocol, the framework tests the edit once on reserved practice problems the session can neither read nor use before the model submits or withdraws.
For the repair task, the researchers overwrite eight consecutive rows of one MLP output matrix (layer 30 of 64 in the 27B model) with noise drawn at 200 times the matrix's standard deviation of entries. The editor is the undamaged model; everything it reads and runs comes from the damaged copy. The practice set is 3,000 math problems from DeepScaleR, filtered to remove overlap with MATH-500, AIME 2024/2025, HMMT February 2025, AMC 2023, OlympiadBench, and GPQA-Diamond; a further 1,000 problems are held out and no session sees them. Evaluations answer each problem with the same seed for base and edited models.
Why This Matters
Impact on research. The paper draws a distinction the field has largely taken for granted: that training and inference are separate phases. It shows a model can act as its own optimizer using only forward passes, and that the resulting gain is written into the weights — paid once, not on every answer, as test-time scaling requires. It also gives the first evidence (to the authors' knowledge) that this ability scales with model size, and it offers a self-repair mechanism that is not a return to stored original weights but the model's own rule. Prior work had a frontier model editing a much smaller one; here the model edits itself.
Real-world applications:
- Self-healing deployed systems. A model could diagnose and patch its own corrupted weights after a hardware or memory fault, without a clean checkpoint or retraining.
- Cheap post-deployment improvement. A model could fix a specific failure mode — such as answers that fail to terminate — from graded feedback, without a training pipeline or cluster.
- Reduced inference cost. Because the improvement lives in the weights, there is no extra computation per answer, unlike harnesses, prompt engineering, or test-time scaling.
- Auditable self-diagnosis. The sessions are recorded in the model's own words and code, producing a step-by-step record from symptom to submitted edit.
Industry relevance. Each session behind the largest gains ran in a few hours on a single GPU, with no pre-training or post-training infrastructure. If improving a model can take a conversation rather than a training run, model maintenance becomes accessible to anyone who can say what goes wrong — and the wall between serving and training becomes a tooling question rather than a hard boundary.
Future Directions
-
Closing the loop across sessions. Each session in the reported protocol starts from the base weights, with each submitted edit replacing the last model in use; the model never begins a session from its own previously edited weights. A true recursive loop — each session starting from the last accepted edit — is the stated aim and remains untested.
-
Why scale is required. The 2B model does not treat its weights as something it can change, and the 4B and 9B models perturb themselves at random. What property of the 27B model enables measurement-driven editing, and whether smaller models could reach it with better tools or more guidance, is open.
-
Generalizing beyond the studied failure modes. The two tasks are a specific verbosity failure and a specific injected weight corruption (eight rows of one MLP output matrix). Whether self-rewriting handles other failure types, other corruption patterns, or non-mathematical domains is not reported.
-
Safety and verification. A model that can write its own weights raises the question of what constrains an edit — which behaviours might be changed or lost alongside the targeted failure — and how accepted edits should be audited beyond the paired tests used here.
Target Audience
Researchers and engineers working on model editing, test-time adaptation, self-improving agents, and post-training; interpretability researchers interested in what directions in hidden states correspond to observable behaviors like non-termination; and practitioners who deploy frozen open-weight models and want post-deployment improvement without training infrastructure. Readers need comfort with transformer architecture and basic statistics to follow the experimental sections, though the session transcripts in the figures make the model's reasoning legible to a broader audience.
Authors’ abstract
A language model is trained once and frozen. Making it better takes another training run, or a prompt, a harness or extra computation at inference that is gone once the task ends. We show a third way: to our knowledge for the first time, a language model rewrites its own weights by its own reasoning and becomes better, with no gradients and no training. We call this self-rewriting: the model is its own optimizer. In earlier work, a frontier model edited a much smaller one;…
The full abstract on arXiv