Transfer Learning and Domain Adaptation

Transfer problems differ. The difference is whether the task, the domain, or both have changed.

The first row of that table has been measured. CAMELYON17-WILDS is built from 455,954 histopathology patches, 96x96 pixels each, and the domain label is nothing more than which of 5 hospitals the patch came from. The underlying CAMELYON17 data come from 5 medical centres in the Netherlands: 1,000 slides from 200 patients. The task is identical everywhere — tumour tissue in this patch or not. Only the centre changes, through staining and acquisition differences between sites. Koh, Sagawa and their co-authors published the benchmark in 2021, and said plainly why they built it: “Models for medical applications are often trained on data from a small number of hospitals, but with the goal of being deployed more generally across other hospitals.” Train on some hospitals, test on a hospital held out of training, and standard empirical risk minimisation scored 93.2% (SD 5.2) average accuracy in-distribution and 70.3% (SD 6.4) out-of-distribution.

That is what “same task, new domain” costs when it is written down as a number instead of a caveat.

A source model, representation, dataset, or task contributes to a target problem; that reuse may take the form of frozen features, partial fine-tuning, initialization, adapters, or shared training.

The benefit depends on which source features remain useful; similar file formats or industries do not guarantee similar predictive relationships.

One assembly line trains a model that appears accurate, but the target line differs in subtle ways. The controlled version of that move exists, with everything but one variable held fixed.

MVTec AD 2 is an industrial inspection dataset: 8 scenarios, 8,004 high-resolution images, and two private test sets. One is shot under the training lighting. The other mixes seen and unseen lighting conditions. The team at MVTec Software GmbH and TU Munich that built it in 2025 says what the second test set is for: “Additionally, our dataset provides test scenarios with lighting condition changes to assess the robustness of methods under real-world distribution shifts.”

Under the training lighting, EfficientAD was the best method at 30.8% mean AU-PRO(0.05). Change the lamps and leave everything else alone — the same parts, the same defects, the same task — and the methods separate. RD lost 1.4 percentage points. MSFlow fell 12.4 points, from 24.3% to 11.9%. Nothing about the inspection problem moved. The light did, and it was worth an order of magnitude more to one method than to another. A factory transfer changes several of these at once.

The amount of target adaptation should match evidence, data volume, and shift severity.

Rung 1 — apply the source model unchanged and measure the gap — is not a formality. The gap can be enormous on a product sold as finished. Epic’s sepsis model was applied unchanged to 27,697 patients across 38,455 hospitalizations at Michigan Medicine. The hospitalization-level AUC was 0.63 (95% CI 0.62-0.64), against the 0.76-0.83 the developer reported. At the recommended alert threshold of 6 the model had 33% sensitivity. It missed 1,709 of the 2,552 patients with sepsis, 67% of them, with a number needed to evaluate of 8. Wong and colleagues put it in one line under “Conclusions and Relevance” in JAMA Internal Medicine in 2021: “This external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis.” A second, unrelated site reached the same verdict. Validating the same model across 145,885 encounters at two county emergency departments, Ostermayer and colleagues reported 14.7% sensitivity and 7.6% PPV, and concluded that “The ESPMv1 provides suboptimal diagnostic characteristics for undifferentiated patients in a county ED.”

Rungs 2 and 4 — frozen features against full fine-tuning — have also been measured against each other rather than argued about. Kumar and Raghunathan, with three colleagues, compared the two across 10 distribution-shift datasets in 2022. Full fine-tuning averaged 2% higher in-distribution accuracy and 7% lower out-of-distribution accuracy than linear probing on frozen features. Their abstract states the finding: “we find that fine-tuning can achieve worse accuracy than linear probing out-of-distribution (OOD) when the pretrained features are good and the distribution shift is large”. Climbing the ladder in order beat jumping to the top rung: linear probing followed by fine-tuning (LP-FT) beat full fine-tuning by 1% in-distribution and 10% out-of-distribution. Wortsman and his co-authors reported the same tension independently at CVPR the same year: “Although existing fine-tuning methods substantially improve accuracy on a given target distribution, they often reduce robustness to distribution shifts.”

Higher on the ladder is not further along. It is a different bet, and the in-distribution number is the wrong place to settle it.

A useful transfer report names both similarity and difference.

A regulator now requires substantially this document before a device reaches the market. The US Food and Drug Administration’s final guidance on predetermined change control plans for artificial-intelligence-enabled device software functions, issued on 4 December 2024 and reissued on 18 August 2025, asks the manufacturer to write the adaptation plan down in advance rather than discover it later. Re-training the model on new data within the intended use population is one of the modification types such a plan may cover. Step 4 of this audit is a filing, not an afterthought.

Step 3 has a regulatory form as well. Section IV of the guidance defines test data: “Test data should be independent of data used for training and tuning and should generally be from multiple sites different from those that were used to generate training and tuning data.” A target-site test set held out of training is what the evidence rests on, in a submission as much as in a project.

Step 6 is anticipated too. The Description of Modifications has to say whether a change is global or is “implemented differently on different devices on the market based on, for example, the unique characteristics of a specific clinical site or individual patients (sometimes referred to as heterogenous or local changes, or local adaptations)”. Per-site adaptation is allowed. Being unable to say which sites were adapted and how is not.

Domain adaptation sometimes encourages source and target representations to look similar, and if the domains differ for a meaningful reason, aggressive alignment can merge classes or remove target-specific signal.

Use target labels where possible and evaluate per class and slice. “Domain invariant” is a design goal, not a universal virtue.

There is a theorem behind that warning, proved in 2019 by Zhao and Tachet des Combes with two colleagues. It is an information-theoretic lower bound, and it characterises “a fundamental tradeoff between learning invariant representations and achieving small joint error on both domains when the marginal label distributions differ”. The construction is not pathological. Two sites with different class balances is the ordinary case in medicine, fraud and manufacturing. Suppose the target site genuinely does have more of one class than the source did. An alignment objective that makes the two look identical is then deleting the very fact the model was brought in to use.

The scoreboard agrees with the theorem. On the same CAMELYON17-WILDS hospital shift, CORAL scored 59.5% (SD 7.7) out-of-distribution accuracy against 70.3% (SD 6.4) for plain empirical risk minimisation. CORAL is a method built for unsupervised domain adaptation by aligning source and target feature distributions, and it finished 10.8 points worse than doing nothing special. IRM reached 64.2% and Group DRO 68.4%, both also below ERM. Gulrajani and Lopez-Paz reached the same verdict from an unrelated direction. They built DomainBed over seven multi-domain datasets, nine algorithms and three model-selection criteria, and their abstract reports that “when carefully implemented, empirical risk minimization shows state-of-the-art performance across all datasets”. Before adopting an alignment objective, make it beat the model that ignores the domain label entirely.