Random Initialization of Gated Sparse Adapters

AI summary

Overview

Research area: Parameter-efficient fine-tuning (PEFT) of large language models, with a focus on sparse adaptation, the Lottery Ticket Hypothesis, and catastrophic forgetting.

Technical level: Intermediate. The paper assumes familiarity with LoRA/QLoRA, low-rank vs. sparse update matrices, and the standard benchmark suite used to measure LLM degradation.

Scope: This paper introduces Random Initialization of Gated Sparse Adapters (RIGSA), a sparse fine-tuning method that generalizes Lottery Ticket Sparse Fine-Tuning to non-zero random initializations and multiple pruning steps, and evaluates it against QLoRA and random masking on a new vision-in-text task while tracking forgetting on three standard benchmarks.

What This Paper Is About

Fine-tuning a language model on a new task tends to damage its performance on tasks it already knew, a problem called catastrophic forgetting. Existing PEFT methods like LoRA limit the size of the update to a low-rank matrix, which constrains how expressive that update can be. This paper asks whether a sparse, full-rank update, initialized randomly and pruned iteratively from a lottery-ticket-style perspective, can let a model learn a new task while forgetting less of what it already knew.

Key Contributions

1. RIGSA: A sparse adaptation method that starts from a randomly-initialized, full-rank difference matrix, gates it with a ReZero-style learnable scalar initialized to 10⁻⁶ so training begins at the pre-trained weights, then sparsifies it with Iterative Magnitude Pruning. The authors describe it as a generalization of Lottery Ticket Sparse Fine-Tuning (LT-SFT) that supports non-zero initializations and multiple pruning steps.

2. Textual MNIST: A new out-of-distribution vision-in-text benchmark in which MNIST images have their pixel values quantized to the [0, 9] range and are rendered as rows of ASCII digits separated by newlines, formatted to match SmolLM2's per-digit tokenization.

3. An empirical comparison: RIGSA is evaluated against 4-bit QLoRA (at ranks 1, 4, 8, and 16) and against a random mask at matched sparsity on both the target task and three source benchmarks.

4. An observation about parameter count and forgetting: RIGSA mobilizes considerably more trainable parameters than QLoRA yet displays less forgetting, a result the authors note runs counter to the expectation that more parameters means more overfitting to the target task.

Main Findings

• The target task is genuinely out-of-distribution for the base model. SmolLM2-1.7B-Instruct scores 9.16% zero-shot and 10.04% 5-shot on Textual MNIST at full precision, and 9.80% / 10.41% in 4-bit quantized form. The slightly sub-chance zero-shot numbers reflect occasional failures to emit a valid digit.

• The task is learnable. After the first RIGSA step (which is effectively dense fine-tuning, since no pruning has occurred yet), the model reaches 99.05% test accuracy on Textual MNIST, despite having over 1.6B free parameters for a dataset of only 60,000 examples. Random masking also learns the task, and QLoRA learns it too.

• Pruning costs target-task accuracy. Accuracy degrades at each subsequent pruning iteration, almost falling to random-mask performance by step 4. The authors hypothesize the high per-iteration pruning ratio (80%) or other hyperparameter settings may be responsible.

• Forgetting is limited on HellaSwag and PIQA but appears on GSM8k. No significant forgetting was observed on HellaSwag or PIQA at any pruning iteration. On GSM8k (5-shot), the dense adapter dropped to 40.7% from the authors' 43.7% baseline, then recovered as it became sparser, reaching 45.1% at step 3. That is still below the 48.8% reported by the SmolLM2 authors. The authors caution they could not run benchmarks multiple times, so they cannot attribute the GSM8k improvement to the method rather than inter-run variability.

• Quantization hits GSM8k hard. The authors' own reproduction gives 43.7% (full precision) and 19.9% (4-bit) on GSM8k, 75.4% and 75.2% on PIQA, and 51.7% and 50.8% on HellaSwag. The authors note their HellaSwag numbers differ from the 66.1% reported in the original SmolLM2 paper.

• QLoRA wins on the target task. QLoRA reaches 99.46% on Textual MNIST at rank 16, versus 98.37% for the sparse adapter and 99.05% for dense fine-tuning. Across trainable-parameter budgets, QLoRA's target accuracy stays relatively stable while RIGSA's rises with the budget.

• RIGSA forgets less than QLoRA. On GSM8k, QLoRA drops from the 19.9% quantized baseline to 14.18% at rank 16 and 11.07% at rank 1. The authors report that RIGSA forgets less than QLoRA on all three benchmarks, despite having more trainable parameters.

• RIGSA does not conclusively beat random masking. The two methods lead to modest and comparable forgetting, and the authors state that a definitive comparison requires more extensive hyperparameter search and multiple benchmark runs.

• The final adapter is very small. After 5 IMP iterations the sparse adapter retains 3.46% of the original trainable parameters.

Methodology in Plain English

The researchers froze every weight of SmolLM2-1.7B-Instruct and trained only a difference matrix added on top of the model. Rather than forcing that matrix to be low-rank like LoRA, they initialized it densely and at random, so that any sparse "winning ticket" hidden inside it could in principle be found.

To avoid destabilizing the model at the start, they multiplied the difference matrix by a single learnable scalar borrowed from ReZero, initialized to 10⁻⁶. Training therefore begins at the pre-trained weights and gradually moves away from them.

After training, they pruned all linear layers except the language modelling head. Following the refinement of Zhou et al. (2019), they kept the 80% of parameters that had not changed sign and had the largest magnitude among those, reset the rest to their initial values, and froze them. They repeated this train-then-prune cycle for 5 iterations, training one epoch per cycle, and trained the final sparse matrix one last time. As a control, they trained an adapter with a random mask at the same final sparsity.

All pruning experiments ran on a single RTX 4090 with 24 GB of VRAM, using AdamW with weight decay of 1.0, 1000 warmup steps peaking at 2×10⁻³ followed by decay to zero, and gradient accumulation to reach an effective batch size of 16. The QLoRA comparison used the 4-bit quantized Unsloth model with rank 1/4/8/16, alpha 32, dropout 0.0, learning rate 2e-4, linear scheduler, weight decay 0.1, batch size 16, 1000 warmup steps, and 4 epochs.

For evaluation, they measured target-task test accuracy at each pruning iteration and measured source-task performance on PIQA (3,000 test questions, two answers each), HellaSwag (10K sentence beginnings, four possible endings each), and GSM8k (1,319 validation math problems), using GSM8k 5-shot and HellaSwag and PIQA 0-shot.

Why This Matters

Impact on research. The paper argues that sparse adaptation may carry inherent regularization benefits: a model given far more trainable parameters than QLoRA still forgot less. If that holds up under repeated runs, it reframes the common assumption that constraining parameter count is what protects against forgetting, and suggests the structure of the update matters as much as its size. The paper also contributes Textual MNIST as a deliberately out-of-distribution probe that compact language models fail at near chance level before fine-tuning.

Real-world applications:

• Domain adaptation with limited compute: Adapting a pretrained model to a specialized task (medical, legal, or technical text) without retraining or damaging general capabilities.
• Continual learning on a fixed base model: Learning a sequence of tasks over time while preserving earlier ones, since forgetting is the central concern the method targets.
• Adapter composition and sharing: Producing small, sparsity-bounded adapters that can be distributed and swapped on a shared frozen base model.
• Edge or on-device personalization: Fine-tuning and storing a small, sparse adapter rather than a full model copy, which matters for deployment on constrained hardware. Note that the paper itself does not measure inference memory or latency for the sparse adapter.

Industry relevance. Fine-tuning costs and forgetting are practical obstacles for teams that build products on top of pretrained models. The paper's framing of pretrained weights as an expensive shared investment, and its goal of building downstream applications with less data and less computation than naive alternatives, speaks directly to that concern. The result that RIGSA uses more parameters yet forgets less than QLoRA is the kind of trade-off a practitioner choosing an adaptation strategy would want to understand.

Future Directions

• Rerun with proper statistics. The authors explicitly call for multiple runs of GSM8k, HellaSwag, and PIQA, since without them they cannot separate their method's effect from measurement noise, including the unexpected source-task improvements seen with random masking.
• Tune the masking ratio. They suggest exploring higher masking ratios, among other hyperparameter searches, to match the trainable parameter counts of common (Q)LoRA configurations and make the comparison fairer.
• Direct comparison with LT-SFT. A head-to-head comparison is suggested, since RIGSA is presented as a generalization of that method.
• Compare against full-precision LoRA. The current comparison is against 4-bit QLoRA only, and quantization is shown to have a large effect on GSM8k specifically (43.7% to 19.9%), so the picture might change against single-precision LoRA.
• Explain the pruning degradation. The drop in target-task accuracy across pruning iterations is attributed speculatively to the 80% per-iteration pruning ratio or other hyperparameters, which remains an open question.

Target Audience

Researchers and engineers working on parameter-efficient fine-tuning, sparse neural networks, or the Lottery Ticket Hypothesis, particularly those interested in how adaptation methods trade target-task accuracy against catastrophic forgetting. It is also relevant to practitioners who need to adapt a pretrained model on modest hardware, since the paper reports running all pruning experiments on a single 24 GB GPU. Readers without background in PEFT or sparsity will need to read the related work section first to follow the distinction between low-rank and sparse updates.

Authors’ abstract

When fine-tuning language models on new tasks, catastrophic forgetting -- performance degradation on previously-learned tasks -- is a ubiquitous problem. While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA address this through low-rank adapters, sparse adaptation offers an alternative that doesn't impose rank constraints. We introduce Random Initialization of Gated Sparse Adapters (RIGSA), which starts from randomly-initialized full-rank adapters, gates them with a ReZero analog, and sparsifies them with iterative magnitude pruning. We evaluate RIGSA on SmolLM2-1.7B-Instruct using a novel vision-in-text task (Textual MNIST) and measure forgetting on PIQA, HellaSwag, and GSM8k. SmolLM2-1.7B-Instruct initially performs around chance level on Textual MNIST, and is capable of learning the task through RIGSA, 4-bit QLoRA and random masking. In spite of having more trainable parameters than QLoRA, the RIGSA configurations that we studied displayed less forgetting than QLoRA, particularly on GSM8k, though it performs comparably to random masking.