Transfer Learning and Fine-Tuning Schedules

Start with the smallest change that can satisfy the target task. The rungs of that ladder have been measured against each other, on GLUE with BERT-base. A frozen encoder, training 0.0% of the parameters, averaged 62.1. Tuning only the query biases, 0.01% of the parameters, reached 76.6. Tuning every bias term, 0.09%, reached 82.4. Full fine-tuning, 100% of the parameters, reached 82.3.

The fraction is the point. In 2022 Ben-Zaken and colleagues put it plainly: “The bias terms are additive, and correspond to a very small fraction of the network, in BERTBASE and BERTLARGE bias parameters make up 0.09% and 0.08% of the total number of parameters in each model, respectively.”

Roughly one thousandth of the parameters matched what moving all of them achieved on that benchmark. That is why the three regimes below are worth choosing between, rather than defaulting through.

When a specialist moves to a clinic with different equipment and patient patterns, the expertise arrives intact. Existing skill is valuable, but some routines require careful adaptation.

Where the analogy parts company with fine-tuning is that a colleague can name which routines to relearn. Parameter groups cannot. Layerwise decay assumes the earlier layers hold the transferable competence. That is a claim about one architecture and one domain shift, not a property of pretrained models. The sections below show what happened when people tested it, by cutting real networks at every layer and reading off what the cut cost.

Phases separate head initialization from backbone adaptation, and each regime has a measured price. Table 7 of the ULMFiT paper reports validation error for four regimes on the same three datasets — IMDb, TREC-6 and AG. Training from scratch: 9.93 / 13.36 / 6.81. Fine-tuning only the last layer ('Last'): 6.49 / 16.09 / 8.38. Full fine-tuning: 6.87 / 6.86 / 5.81. Gradual unfreezing with discriminative rates and slanted triangular rates ('Freez + discr + stlr'): 5.00 / 5.69 / 5.38.

Head-only adaptation was the worst of the four on two of the three tasks. It lost even to training from scratch on TREC-6, 16.09 against 13.36, and on AG, 8.38 against 6.81. Howard and Ruder say why: “'Last', the standard fine-tuning method in CV, severely underfits and is never able to lower the training error to 0.”

The order of the phases below is itself a borrowed claim. They unfreeze “starting from the last layer as this contains the least general knowledge (Yosinski et al., 2014)”.

Compare regimes under matched target evidence and selection rules. Steps 5 and 6 deserve the same standing as the in-domain score.

Step 5 has a documented payoff. Probing first and then fine-tuning — LP-FT — was 1% better in-distribution and 10% better out-of-distribution than full fine-tuning, in the same study by Kumar and colleagues. That is the staged ordering the timeline above already describes.

Step 6 needs more than one run per arm. Chen and colleagues fixed their protocol at 20 runs for each low- and medium-resource task and 10 for each high-resource one, across 12 GLUE and SuperGLUE tasks. It took that many runs to see that the parameter-efficient methods led only in the low-resource regime.

Transfer has to be judged on more than one in-domain score.

A pretrained model encodes representations shaped by its source data and objective. Fine-tuning asks those parameters to support a new target, distribution, and decision.

Good source performance does not guarantee useful transfer. Measure target baselines, domain distance, and whether frozen features already contain the required evidence.

ULMFiT put numbers on that in 2018. Howard and Ruder's recipe added discriminative fine-tuning and gradual unfreezing to a pretrained language model. On six text classification tasks it cut error by “18-24% on the majority of datasets”. With “only 100 labeled examples” it matched training from scratch on 100x more data. The pretrained starting point was worth roughly two orders of magnitude of labels.

Lower rates in earlier layers assume those features are more transferable and should change less. The rule has a constant in it, and the constant was tuned, not derived. Howard and Ruder set each layer's rate from the layer above: “We empirically found it to work well to first choose the learning rate ηL of the last layer by fine-tuning only the last layer and using ηl−1 = ηl/2.6 as the learning rate for lower layers.”

That constant has since been swept end to end, and the table published. On IMDb with BERT, at base rate 2.0e-5, ξ=0.95 was best at 5.40% test error. ξ=1.00 — no decay at all — gave 5.42%. ξ=0.85 gave 5.65%. The whole sweep spans 0.25 points. Sun and colleagues' recommendation, in 2019: “We find that assign a lower learning rate to the lower layer is effective to fine-tuning BERT, and an appropriate setting is ξ=0.95 and lr=2.0e-5.”

Where the network is cut matters more than how steeply the rates decay below the cut. Yosinski and colleagues took an eight-layer ImageNet network and cut it at each of layers 1 through 7. Freezing at layers 3–6, and particularly at 4 and 5, lost top-1 accuracy, even when the copied weights came from a network trained on the same task. The boundary splits neurons that depend on each other: “This performance drop is evidence that the original network contained fragile co-adapted features on successive layers, that is, features that interact with each other in a complex or fragile way such that this co-adaptation could not be relearned by the upper layers alone.”

Transferring features and then fine-tuning them still beat training on the target directly, by a mean of 1.6% across layers 1–7 and 2.1% when at least five layers were kept. So inspect layerwise updates and target gains. A decay factor is not a universal rule.