The Surprising Effectiveness of Deleting Weights in LLM Post-Training

AI summary

Overview

Research area: Machine learning — large language model post-training, parameter-efficient adaptation, and the geometry of fine-tuning.

Technical level: Intermediate. The paper assumes familiarity with fine-tuning methods (full fine-tuning, LoRA) and reinforcement learning from verifiable rewards, but its core idea — learning a binary keep/delete mask — is easy to grasp.

Scope: The paper shows that learning only which pretrained weights to set to zero, without ever writing a new weight value, matches full fine-tuning across reinforcement learning with verifiable rewards (RLVR), supervised fine-tuning (SFT), and on-policy distillation (OPD), while producing far smaller adapters and forgetting much less.

What This Paper Is About

Fine-tuning an LLM normally means changing parameters — all of them in full fine-tuning, a low-rank update in LoRA, or float values at selected positions in sparse methods. This paper asks how little of a pretrained model actually has to change for it to acquire new capabilities, and pushes that question to an extreme: can a model learn a new task by only deleting weights, never writing new ones? The authors propose Deletion Tuning, which learns a binary mask over the gate projection of transformer blocks so that each target weight either keeps its pretrained value or is zeroed out.

Key Contributions

1. Deletion alone matches full fine-tuning. Deletion Tuning learns only which gate-projection weights to delete, and it is on par with full fine-tuning at the largest scale tested in each of three regimes — GRPO, SFT, and SDFT — up to a 30B mixture-of-experts model. Under GRPO it also solves problems that the base model fails in all 128 sampled attempts.
2. Compact adapters and less forgetting. The trained masks are hundreds to thousands of times smaller than FFT checkpoints in all three regimes, and Deletion Tuning keeps prior-task accuracy near the base level throughout training.
3. FFT-like update geometry. The learned updates of Deletion Tuning resemble full fine-tuning's more closely than LoRA or random sparse fine-tuning on three measures — stable rank, spectrum drift, and principal-weight overlap — offering an explanation for its accuracy.
4. Evidence that the gains come from learning the mask. Applying the same Wanda one-shot pruning prior without training stays within 2.4 points of the base model on all six GRPO backbones, so the improvement is attributable to mask learning rather than the initialization prior.

Main Findings

• Deletion is extremely sparse. Zeroing fewer than 0.05% of a model's weights matches full fine-tuning at scale. Every GRPO run deletes fewer than 0.05% of all weights; measured deletion rates were 0.035% to 0.046% at the Llama scales studied for retention.
• Parity at the largest backbones. Under GRPO, Deletion Tuning scores 83.55% versus FFT's 83.24% on Llama8B GSM8K and 52.32% versus 51.89% on Qwen7B SimpleRL-Zoo. It also matches FFT on Qwen14B (55.43% versus 54.87%) and on the Qwen3-30B-A3B mixture-of-experts model (58.54% versus 58.43%), where the gate projection inside each of 128 experts per layer is masked and the router is left unchanged.
• The gap closes with scale. At smaller backbones Deletion Tuning recovers 90% to 99% of FFT's accuracy — for example 54.44% versus 60.50% on Llama1B GSM8K and 17.64% versus 19.01% on Qwen0.5B — with parity appearing at the larger scales.
• New capabilities, not just preservation. On pass@k estimated from 128 samples per problem at temperature 1.0, Deletion Tuning leads the base model at every k. At k=128 the base model still trails by 4.3 points on Llama8B and 6.1 points on Qwen7B.
• Smallest adapter at every scale. Adapters run from 0.5 MB at Qwen0.5B to 8.4 MB at Llama8B, 3,600 to 4,700 times smaller than FFT checkpoints, and are smaller than LoRA r=1 in the main GRPO comparison. After zstd compression the mask takes about 0.03 bits per target weight, while every other method stores floating-point values.
• Forgetting is much lower. By step 348, FFT's forward KL reaches 1.0 on Llama1B and 1.5 on Llama8B, and its prior-task accuracy ends 1.3 to 3.1 points below base. Deletion Tuning's forward KL stays at or below 0.05 at all three Llama scales, with prior-task accuracy within one point of base throughout. LoRA and MTH, despite training only a small fraction of parameters, reach forward KL of 0.4 to 1.3 and lose up to 5.4 points on the six-task prior suite (HellaSwag, MMLU, TruthfulQA, WinoGrande, IFEval, HumanEval).
• Sparsity level alone does not explain retention. MTH at 99.95% sparsity trains 0.05% of all parameters, comparable to the 0.035% to 0.046% Deletion Tuning deletes, yet MTH forgets far more — suggesting deletion itself, not the small number of edits, keeps forgetting low.
• Update geometry tracks FFT. Deletion Tuning's stable rank of 66 to 93 sits far below MTH's 410 to 650 and above LoRA's 1 to 2. Its spectrum drift stays within 1.4x of FFT's, while LoRA r=1 drifts up to 84x more than FFT on Llama8B. Its principal-weight overlap of 0.43 to 0.46 matches FFT's sub-random 0.41 to 0.46, versus LoRA at 0.48 to 0.50 and MTH at 0.50.
• Works beyond reinforcement learning. Under SFT, Deletion Tuning matches FFT at Llama3B on MMLU-Pro (22.28% versus 21.19%) and exactly ties on BBEH (11.39% for both); at Llama1B it beats FFT on MMLU-Pro (18.62% versus 16.97%) but trails on BBEH (5.65% versus 8.35%). Under SDFT on Qwen7B it scores 69.07% versus FFT's 68.04% on ToolAlpaca and 70.61% versus 71.20% on Science.
• Robust to hyperparameters. Sweeps over initial sparsity, learning rate, and regularization strength land within 1.6 points of the best run on Llama8B and 3.2 points on Qwen7B; varying the initial sparsity 4x changes accuracy by at most 0.8 and 1.7 points respectively.
• The soft/hard mask gap is negligible. Deploying the hard mask instead of the annealed soft mask changes accuracy by at most 0.08 points.

Methodology in Plain English

The setup. Take a pretrained model and freeze it completely. For one weight matrix per transformer block — the gate projection of the MLP — attach a trainable "logit" to every single weight, of the same shape as the weight. During training, each weight is multiplied by a sigmoid of its logit times a sharpness factor, so every weight gets a soft multiplier between 0 and 1. The logits are the only things that get updated.

Making it binary. The sharpness factor starts small (10³) and ramps linearly up to 10⁴ over training, progressively squeezing the soft multipliers toward 0 or 1 — a process the authors call mask sharpening. At deployment the multiplier is a hard threshold: positive logit means keep the pretrained weight, negative logit means set it to zero. The adapter is therefore one bit per masked weight.

Initialization. Logits are not initialized randomly. The authors compute the Wanda one-shot pruning score (weight magnitude times input activation norm) on calibration prompts from the task distribution, then sample logits from a Gaussian centered at +0.03 for high-importance weights and −0.03 for low-importance ones, split at the η-quantile. At the default η = 1/2048 this deletes roughly 0.18% of gate-projection weights at initialization.

Training objective. The loss is the regime's own objective — cross-entropy for SFT, the GRPO clipped policy-gradient loss, or the forward KL to an EMA teacher for SDFT — plus a sparse-weight regularizer of −λ‖W̃‖²₂ borrowed from MaskLLM that mildly biases toward retaining larger weights. The sparsity level is not fixed in advance; training decides how many weights to delete. In the on-policy regimes, rollouts use the hard mask while gradients flow through the soft mask.

Experiments. Two model families (Llama and Qwen), eight backbones from 0.5B to 30B. GRPO is tested in-distribution on GSM8K and out-of-distribution on the SimpleRL-Zoo setup with a six-benchmark mean (GSM8K, MATH500, Minerva Math, OlympiadBench, AIME24, AMC23). SFT is tested per-subject on 14 MMLU-Pro subjects and 23 BBEH subtasks. SDFT is tested on ToolAlpaca tool use and the Chemistry L-3 subset of SciKnowEval. Baselines are full fine-tuning, LoRA at ranks 1 and 4, and MTH random sparse fine-tuning at 99% and 99.95% sparsity, all with the same data and step counts.

Diagnostics. To explain the results, the authors compare deployed weight deltas against the pretrained weights using stable rank, normalized singular-value spectrum drift, and the fraction of changed positions landing on the base model's "principal weights" (the top 50% by magnitude of a rank-128 approximation). They also track forward KL on the new task as a predictor of forgetting, following prior work.

Why This Matters

Impact on research. The paper reframes post-training as a selection problem rather than a writing problem: capability can be acquired by choosing which pretrained weights to remove, without ever introducing a new nonzero value. That is a strong statement about how much headroom pretrained models already contain, and it connects two previously separate literatures — pruning, which preserves existing capability, and adaptation, which adds new capability. The geometric analysis also gives a mechanistic account of why a binary update can mimic a dense one.

Real-world applications:

• Multi-task serving with cheap adapter switching. Because adapters are 0.5 MB to 47.1 MB depending on model and regime, swapping a task means loading a compact bitmask and scattering zeros into the base weights, not loading a separate full checkpoint.
• Continual and sequential learning. Keeping prior-task accuracy within one point of base while learning new tasks directly addresses catastrophic forgetting, which matters for models that are repeatedly updated.
• Storage- and bandwidth-constrained deployment. A 0.5 MB Llama1B mask versus a 4,714 MB FFT checkpoint changes what is feasible for on-device or edge updates.
• Interpretability and task localization. Comparing which weights different tasks delete could reveal where post-training changes actually live in a model.

Industry relevance. The savings are large and concrete: 3,600 to 4,700 times smaller artifacts than full checkpoints in the main GRPO comparison, plus lower training memory than full fine-tuning (though higher than LoRA, since one logit per masked weight is retained during training). For organizations maintaining many fine-tuned variants of a base model, a bitmask-per-task scheme is far cheaper to store, ship, and hot-swap.

Future Directions

• Sequential continual learning. Since forgetting is so low, the natural next test is training over a sequence of tasks rather than a single one — the paper explicitly names this as future work but does not report such experiments.
• Reducing training memory. Training keeps one logit per masked weight, so Deletion Tuning uses less memory than full fine-tuning but more than LoRA. Restricting logits to a small candidate set of weights could close that gap.
• Mask merging. Masks for different tasks might be combined — for example by taking their union — to merge skills without further training; this is proposed but untested in the paper.
• Structured masks. Constraining the mask to N:M, channel-wise, or block patterns could let a single mask both adapt the model and compress it for efficient inference.

Target Audience

Researchers and engineers working on LLM post-training, parameter-efficient fine-tuning, and model compression will get the most from this paper, particularly those interested in continual learning and catastrophic forgetting. It is also useful for practitioners who need to serve many task variants of a large base model under tight storage or bandwidth budgets. Readers should be comfortable with standard fine-tuning baselines and with reinforcement learning for LLMs; the masking mechanism itself is explained from first principles and is accessible to anyone who knows what a sigmoid is.

Authors’ abstract

How little of a pretrained large language model (LLM) has to change for it to acquire new capabilities? We find that zeroing fewer than 0.05% of its weights, with no other modification, matches full fine-tuning (FFT) at scale across three LLM post-training regimes: reinforcement learning with verifiable rewards, supervised fine-tuning, and on-policy distillation. Our method, Deletion Tuning, learns a binary mask over the gate projection of transformer blocks. Two practical…