Learned Visual Representations and Transfer

A pretrained encoder may separate common objects well while ignoring microscopic texture, thermal patterns, or rare industrial defects. What it gives you depends on what the pretraining objective preserved.

Transfer learning is a claim about four things: a source model, target data, an adaptation method, and an evaluation boundary. It is also a claim that can be put on a scale. A 2019 paper called Transfusion put it on one, using a retinal fundus task. A ResNet-50 trained from random initialization reached 96.4% ± 0.05 AUC. The same architecture initialized from ImageNet reached 96.7% ± 0.04. Those are the two numbers; the gain pretraining bought sits between them. The authors' own small model, CBR-Tiny, has 1,076,480 parameters and an ImageNet top-5 accuracy of 73.5%, against ResNet-50's 92%. It scored 95.8% with ImageNet initialization and 95.8% without. Raghu and colleagues state the finding without hedging: “A performance evaluation on two large scale medical imaging tasks shows that surprisingly, transfer offers little benefit to performance, and simple, lightweight models can perform comparably to ImageNet architectures.”

Generality also stops somewhere, and where it stops has been measured. In 2014 Yosinski and colleagues transferred features layer by layer out of an ImageNet-trained network. They report “two distinct issues: (1) the specialization of higher layer neurons to their original task at the expense of performance on the target task, which was expected, and (2) optimization difficulties related to splitting networks between co-adapted neurons, which was not expected”. They also document “that the transferability of features decreases as the distance between the base task and target task increases, but that transferring features even from distant tasks can be better than using random features”. Distance costs you. It rarely costs you everything.

Each adaptation level trades flexibility, data needs, compute, and risk of forgetting. The distance between two levels is measurable rather than notional. Take one task and one architecture: a subset of PASCAL VOC 2011 segval, scored in the fully convolutional networks paper. A frozen backbone with only its last layer trained, FCN-32s-fixed, gave 45.4 mean IU. Fine-tuning the same network end to end at the same coarse stride gave 59.4. Two rungs of one ladder: 45.4 against 59.4. Which rung a project belongs on is settled by running them, not by preference.

The best choice depends on target data, mismatch, compute, and deployment constraints. The trade has been quantified. Full fine-tuning was compared against linear probing across 10 distribution-shift datasets: Breeds-Living17, Breeds-Entity30, DomainNet, CIFAR→STL, CIFAR10.1, FMoW, ImageNetV2, ImageNet-R, ImageNet-A and ImageNet-Sketch. Averaged over them, fine-tuning was 2% more accurate in-distribution and 7% less accurate out-of-distribution. Kumar and colleagues state the reversal in their 2022 abstract: “However, in this paper, we find that fine-tuning can achieve worse accuracy than linear probing out-of-distribution (OOD) when the pretrained features are good and the distribution shift is large.” The remedy they measured is an order rather than a choice. Linear-probe first, then fine-tune: 1% better in-distribution and 10% better out-of-distribution than full fine-tuning alone.

The tension is not a property of one group's models. Wortsman and colleagues report it for zero-shot models, also in 2022: “Although existing fine-tuning methods substantially improve accuracy on a given target distribution, they often reduce robustness to distribution shifts.” So the row that reads risk: unstable training is really two numbers with opposite signs. Which one a deployment feels depends on how far its data drifts.

Transfer can fail even when the target images look perfectly familiar to a human. The failure has been caught in the act. Pneumonia-screening CNNs were trained on 158,323 chest radiographs from three hospital systems and the result was published in PLOS Medicine on 6 November 2018. Internally the best model reached AUC 0.931 (95% CI 0.927–0.936). At the external Indiana University site it fell to AUC 0.815 (95% CI 0.745–0.885, P = 0.001). The same abstract names the confounder the networks were exploiting: “CNNs were able to directly detect hospital system of a radiograph for 99.95% NIH (22,050/22,062) and 99.98% MSH (8,386/8,388) radiographs.” Zech and colleagues had a model that could tell you where a chest was x-rayed.

The identical mechanism turned up later in a different disease at a different institution. COVID-19 radiograph systems “rely on confounding factors rather than medical pathology, creating an alarming situation in which the systems appear accurate, but fail when tested in new hospitals”. DeGrave and colleagues published that in Nature Machine Intelligence in 2021.

How general is the pattern? WILDS, a 2021 benchmark, assembled 10 datasets of naturally occurring shifts and ran standard empirical-risk-minimization training on every one. Koh and colleagues report the outcome in a single line: “On each dataset, we show that standard training yields substantially lower out-of-distribution than in-distribution performance.” Not most datasets. Each one.

Compare increasingly flexible adaptation levels before committing to a large training run. The first rung is not a formality. Standard COCO detection and segmentation models have been trained from random initialization, and they matched their ImageNet-pretrained counterparts once the training iterations were increased. Detection reached 50.9 AP on COCO with no external data. He and colleagues state the conclusion: “Experiments show that ImageNet pre-training speeds up convergence early in training, but does not necessarily provide regularization or improve final target task accuracy.” A separate group reached a compatible result independently in 2020: on COCO, ImageNet pre-training “benefits when we use one fifth of the labeled data, and hurts accuracy when we use all labeled data”. A from-scratch baseline that is never run is a comparison decided in advance.

Step 5 is where the WILDS figures belong. A single pooled score cannot hold a 22.9-point in-distribution-to-out-of-distribution gap. Only the slice can. That is why the last rung measures domain conditions, rare cases, latency, memory, and stability rather than one headline number.

Early stages often retain local edges and texture, while later stages become more task-specific and spatially compressed. That tendency is not a guarantee across architectures or objectives.

The fully convolutional networks paper measured the choice rather than arguing it. On a subset of PASCAL VOC 2011 segval, in 2015: a frozen backbone with only the last layer trained, FCN-32s-fixed, gave 45.4 mean IU. Fine-tuning end to end at the same coarse stride gave 59.4. Fusing predictions from pool4, FCN-16s, gave 62.4. Adding pool3, FCN-8s, gave 62.7 — and there Long and colleagues report diminishing returns and stop fusing lower layers. What the fusion is for is stated in their abstract: “We then define a skip architecture that combines semantic information from a deep, coarse layer with appearance information from a shallow, fine layer to produce accurate and detailed segmentations.” The same abstract records a “20% relative improvement to 62.2% mean IU on 2012”.

Rankings between layers are fragile to the instrument doing the ranking. Probes on ELMo's first layer beat the second on task accuracy, 97.2 against 96.6, while being markedly less selective, 26.0 against 31.4. Hewitt and Liang's ordering inverts depending on which column you read. Dense prediction may require intermediate spatial features; image retrieval may prefer a global embedding. Validate layer choice on the target task.

A single final score hides whether gains came from pretraining, extra compute, or favorable tuning. Documenting the split is not a house style rule. Three regulators — the U.S. FDA, Health Canada and the UK MHRA — “have jointly identified 10 guiding principles” for Good Machine Learning Practice for Medical Device Development, published on 27 October 2021. Principle 3 is “Clinical Study Participants and Data Sets Are Representative of the Intended Patient Population”. Principle 4 is “Training Data Sets Are Independent of Test Sets”, and it reads: “Training and test datasets are selected and maintained to be appropriately independent of one another.” The principle asks that all potential sources of dependence — patient, data acquisition and site factors among them — be considered and addressed. Three agencies wrote down the leakage control that a transfer evaluation has to be able to show.