Transfer Learning and Feature Reuse

Training from scratch asks the target dataset to teach two things at once: general structure and task-specific behavior. Transfer learning starts somewhere else. It brings in knowledge acquired elsewhere — visual edges and shapes, language regularities, acoustic patterns, user behavior, a representation of related tasks — and the target training process reuses or adapts it. The benefit is strongest when target labels are scarce and the source experience captures useful invariances.

A large source model is not automatically a good source model. If the source rewards shortcuts, ignores target-relevant detail, or represents a substantially different population, transfer can slow learning or cap performance. That is negative transfer, and it is not a loose figure of speech. Pan and Yang's 2010 survey of the field defines it: negative transfer happens “when the source domain data and task contribute to the reduced performance of learning in the target domain”. So treat transfer as a hypothesis. The claim being made is that the source representation reduces target sample complexity or improves target robustness. The target validation protocol is the judge of it.

Where the general part ends and the specific part begins can be measured. In 2014 Yosinski and three co-authors split ImageNet’s 1,000 classes into two halves of 500, roughly 645,000 examples each, and trained eight-layer networks on both. Features, they report, “must eventually transition from general to specific by the last layer of the network.” Transfer still paid off after fine-tuning. The average boost across layers one to seven was 1.6%, and 2.1% when at least five layers were kept.

Relatedness is multidimensional. A source can match in one way and differ in another. Which dimension actually pays? Kornblith and colleagues put numbers on that question in 2019, running 16 classification networks across 12 image classification datasets. Fine-tuning beat random initialization on 189 of the 192 dataset/model combinations. On two fine-grained sets the margin nearly disappeared. Inception v4 at 299x299 scored 92.7% on Stanford Cars from scratch against 93.3% fine-tuned, and 88.8% on FGVC Aircraft from scratch against 89.0% fine-tuned.

The diagnosis is semantic, not one of scale. The ImageNet training set holds more car images than Stanford Cars does — 12,455 against 8,144 — but sorts them into only 10 high-level car classes. Stanford Cars uses 196 classes, by make, model and year. More source data about cars, carrying none of the distinctions the target labels need. Their conclusion: “ImageNet pretraining thus appears to have only marginal accuracy benefits for fine-grained classification tasks where labels are not well-represented in ImageNet.” Their headline result does not rescue those two cases. Measured across all the datasets, ImageNet accuracy correlates with transfer accuracy at r = 0.99 for fixed feature extractors and 0.96 for fine-tuning. Aggregate relatedness and the relatedness your labels need are different quantities.

Transfer learning is broader than copying neural-network weights. Four different things can move, and the four boxes here are not a house categorisation. They are the four published cases, from Pan and Yang’s 2010 survey in IEEE Transactions on Knowledge and Data Engineering. The survey puts it this way: “Approaches to transfer learning in the above three different settings can be summarized into four cases based on “What to transfer”.”

Table 3 of that survey defines each case in a line. Instance-transfer: “To re-weight some labeled data in the source domain for use in the target domain”. Feature-representation-transfer: “Find a 'good' feature representation that reduces difference between the source and the target domains”. Parameter-transfer: “Discover shared parameters or priors between the source domain and target domain models”. Relational-knowledge-transfer: “Build mapping of relational knowledge between the source domain and the target domains”.

Naming the case is not pedantry. The object you transfer decides what can go wrong with it. Reweighted instances import the source population. A reused feature representation imports the source’s distinctions. Shared parameters import the source’s inductive bias. Transferred relational structure imports the source’s assumptions about how entities relate.

A credible transfer study compares more than one adaptation depth, and the frozen middle option is the one with the strongest published pedigree. Razavian and three co-authors established it in 2014. They took the OverFeat network trained for object classification on ILSVRC13 and froze it. Out of a single layer came a 4096-dimensional representation, with a linear SVM on top — L2 distance in the retrieval case. The datasets were picked to “gradually move further away from the original task and data the OverFeat network was trained to solve”. They run from object image classification through scene recognition, fine-grained recognition and attribute detection to image retrieval. That frozen pipeline beat highly tuned state-of-the-art systems on every visual classification task tested. Their closing sentence: “The results strongly suggest that features obtained from deep learning with convolutional nets should be the primary candidate in most visual recognition tasks.”

So the frozen encoder is not a formality on the way to the real experiment. It is the baseline a fine-tuning result has to beat before the extra parameters, the extra compute and the extra forgetting risk are justified. And the scratch model is the baseline that says whether the source was needed at all.

The experiment should reveal where the gain comes from and when it disappears. That design has been run at full scale, and the steps below are worth reading against what it actually did.

In 2018 Zamir and five co-authors took 26 2D, 2.5D, 3D and semantic tasks and trained every one of them on the same pixels. The pixels came from a purpose-built dataset: 4 million images of indoor scenes from about 600 buildings, annotated for every task. A difference between two tasks could not be a difference between two datasets. That is step 1 taken seriously.

Step 5 is in their protocol as well. The transfer networks were deliberately trained on 1,000 to 16,000 images “in order to model the transfer patterns under different data regimes”, which is a data-size curve rather than a single number. Measuring relatedness instead of assuming it paid for itself. Their abstract: “the total number of labeled datapoints needed for solving a set of 10 tasks can be reduced by roughly 2/3”, at nearly unchanged performance. The map, not the intuition, bought the labels back.

The right amount of adaptation depends on target evidence, representation mismatch, and operational cost. The middle column now has a price tag rather than an adjective. In 2019 Houlsby and seven co-authors transferred BERT to 26 diverse text classification tasks using adapter modules, with the pretrained network’s own parameters left frozen. Their abstract reports: “On GLUE, we attain within 0.4% of the performance of full fine-tuning, adding only 3.6% parameters per task. By contrast, fine-tuning trains 100% of the parameters per task.”

Read the columns with those two figures in hand. Full adaptation buys the last 0.4% on that benchmark at 100% of the parameters per task. Partial adaptation gives most of the quality back for 3.6%. Whether the remaining fraction of a point is worth a separate copy of the model for every task is a target-domain and governance question. It is answerable only after the frozen probe and the partial-adaptation run have both been measured.