Fine-Tuning and Parameter-Efficient Adaptation

A pretrained model contains a hierarchy of reusable and source-specific behavior. Adaptation can update a small output head, the upper blocks, all parameters, or a compact set of inserted parameters. Each of those choices changes sample efficiency, memory use, storage, and the risk of overwriting useful knowledge. Full fine-tuning offers maximum freedom, but it can be wasteful or unstable when target data is small. Partial fine-tuning preserves more of the source representation. Parameter-efficient methods freeze most pretrained weights and learn a smaller set of task-specific parameters, which is attractive when many tasks share one base model. The right method depends on more than training cost. It depends on how far the target domain is from pretraining, on whether multiple task variants must coexist, and on whether deployment can load separate full checkpoints.

Adapter modules put a number on the compact end. Neil Houlsby and seven colleagues transferred BERT to 26 diverse text classification tasks with them in 2019. “The parameters of the original network remain fixed”, and only the inserted ones train. On GLUE those adapters added “only 3.6% parameters per task”, and new tasks “can be added without revisiting previous ones”. By contrast “fine-tuning trains 100% of the parameters per task”. Twenty-six tasks, one frozen base, and nothing to revisit when the twenty-seventh arrives.

A useful experiment orders these strategies from least invasive to most flexible. Two published measurements fix the ends of that order, and neither end wins outright.

A 65-billion-parameter model can be finetuned on a single 48GB GPU. That is the compact end priced in hardware. Tim Dettmers and three colleagues backpropagated gradients through a frozen 4-bit quantized model into LoRA adapters, and published QLoRA at NeurIPS in 2023. Their abstract opens: “We present QLoRA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance.” The Guanaco family that came out of it reached 99.3% of ChatGPT's performance level on the Vicuna benchmark, after 24 hours of finetuning on one GPU. More than 1,000 models were finetuned for the study.

The flexible end keeps an advantage the compact end does not erase. LoRA and full finetuning were compared head to head on programming and mathematics. Instruction finetuning on ~100K prompt-response pairs, continued pretraining on ~20B tokens. The result appeared in Transactions on Machine Learning Research in August 2024: “Our results show that, in the standard low-rank settings, LoRA substantially underperforms full finetuning. Nevertheless, LoRA better maintains the base model's performance on tasks outside the target domain.” Dan Biderman and eleven colleagues ran it. LoRA also forgot less than weight decay or dropout, and the explanation they give is dimensional: full finetuning learned perturbations of rank 10–100× greater than typical LoRA configurations. So the compact end is not simply the cheap version of the flexible one. It learns less and it forgets less. Which half of that sentence matters depends on whether the product still needs the capabilities outside the target domain.

Lower layers often receive smaller updates because they may encode more general structure. That is usually presented as a design intuition. It was in fact written down as a rule, with a divisor and a schedule, and measured.

The divisor is 2.6. Each layer gets a learning rate 2.6 times smaller than the layer above it. Jeremy Howard and Sebastian Ruder specified that in ULMFiT at ACL 2018, along with the pace at which layers are opened. Discriminative fine-tuning, in their section on “Target task LM fine-tuning”, first chooses the learning rate ηL of the last layer by fine-tuning only that layer, then uses η(l−1) = η(l)/2.6 for each layer below it. Gradual unfreezing, a paragraph inside “Target task classifier fine-tuning”, unfreezes the last layer, fine-tunes all unfrozen layers for one epoch, then unfreezes the next lower frozen layer and repeats. The stated reason is the failure mode this whole lesson circles: “Rather than fine-tuning all layers at once, which risks catastrophic forgetting, we propose to gradually unfreeze the model starting from the last layer as this contains the least general knowledge” — Howard and Ruder, section “Gradual unfreezing”.

The schedule bought something measurable. On six text classification tasks the method outperformed the state of the art, “reducing the error by 18-24% on the majority of datasets”. And “with only 100 labeled examples, it matches the performance of training from scratch on 100× more data”. A divisor of 2.6 per layer and one further layer opened per epoch remain design choices rather than laws. But they are published choices with numbers attached. That is a different object from a hunch.

Fine-tuning failures often look like ordinary validation noise until the right comparisons are present.

One of those comparisons inverts. Over ten distribution-shift datasets, fine-tuning averaged “2% higher accuracy ID but 7% lower accuracy OOD than linear probing”. Ananya Kumar and four colleagues measured that at ICLR 2022, and named the mechanism: the lower layers “change simultaneously and distort the pretrained features”. Probing first and then fine-tuning recovered both ends. That order was “1% better ID, 10% better OOD than full fine-tuning”.

Seed variance has a number too, and the explanation usually attached to it is wrong. BERT was fine-tuned with 25 random seeds per dataset. Under the default recipe of Devlin et al. (2019), the development-set standard deviation was 25.6 Matthews correlation points on CoLA (mean 45.6, maximum 64.6), 4.5 points on RTE (mean 50.9, max 67.5) and 3.9 on MRPC (mean 84.0, max 91.2). Marius Mosbach and two colleagues ran those seeds for ICLR 2021. Their abstract states the problem plainly: “Despite the strong empirical performance of fine-tuned models, fine-tuning is an unstable process: training the same model with multiple random seeds can result in a large variance of the task performance.” Their revised recipe cut those standard deviations to 1.8, 2.7 and 0.8 respectively, significant at p < 0.001 by Levene's test. And the cause was neither catastrophic forgetting nor a small target dataset. It was vanishing gradients in the lower layers. A “failed run”, in their definition, is one ending at or below a majority classifier — which is the shape a large standard deviation actually takes on a results table.

Run the ladder with the same target splits and the same regression suite throughout. Each rung has to beat the rung below it on four measurements: target slices, retained capabilities outside the target domain, the spread across random restarts, and the cost of storing and serving whatever it produces. Beating the rung below on the aggregate target metric that motivated the climb is not enough.