Contrastive Adapter Training and Consensus Knowledge Distillation for Multi-Source-Free Domain Adaptation in Skin Cancer Diagnosis

AI summary

Overview

• Research area: Computer vision for medical imaging; specifically multi-source-free unsupervised domain adaptation (MSFDA) applied to skin cancer diagnosis (melanoma versus benign nevi), with vision transformers, parameter-efficient fine-tuning, and knowledge distillation.
• Technical level: Intermediate to Advanced. The paper assumes familiarity with vision transformers (ViTs), domain shift, contrastive learning, adapters/LoRA-style tuning, pseudo-labeling, and knowledge distillation via KL divergence.
• Scope: The paper proposes CAT-CKD, a two-stage framework that builds a shared domain-invariant ViT backbone using contrastively trained convolutional adapters over frozen weights, then adapts a student model to an unlabeled target clinical dataset by distilling an averaged "consensus" of multiple source-model logits — all without access to source images during adaptation.

What This Paper Is About

Deep models that distinguish melanoma from benign nevi often degrade when deployed on images from a different clinical center, because imaging devices, acquisition protocols, and patient demographics differ. Fixing this normally requires access to the original source data, which privacy rules and institutional policies typically forbid once a model has been deployed. The paper's goal is to adapt a model to a new, unlabeled skin-lesion dataset using only already-trained source models — no source images — while avoiding the overfitting, catastrophic forgetting, and noisy supervision that the authors attribute to existing MSFDA methods.

Key Contributions

1. CAT-CKD framework: An MSFDA pipeline for skin cancer diagnosis that explicitly builds a shared domain-invariant backbone before any source-specific models are trained, targeting both representation-level and supervision-level weaknesses of prior methods.
2. Contrastive Adapter Training (CAT): A parameter-efficient strategy that inserts ConvPass convolutional bypass adapters into a frozen ViT backbone and trains only those adapters with supervised contrastive learning (SCL) across all source domains, with domain- and class-weighted sampling to counter imbalance.
3. Consensus Knowledge Distillation (CKD): A distillation strategy that averages the logits of multiple trained source models into a single soft consensus target and trains a student on unlabeled target data with a KL-divergence objective — presented with a variance-reduction argument (Lemma 1) and an optimum-matching argument (Lemma 2).
4. Empirical and efficiency evaluation: Experiments, ablations, and efficiency analyses across five public skin lesion datasets, reporting an average AUROC of 86.1% with only 4.3M trainable parameters, plus a public code release.

Main Findings

• Headline performance: CAT-CKD achieves an average AUROC of 86.1% across the five datasets, which the authors state outperforms existing MSFDA methods — while operating without source data during adaptation.
• Parameter efficiency: The method requires only 4.3M trainable parameters, which the authors attribute to training adapters, bottlenecks, and heads on a frozen ViT-B/16 backbone rather than fully fine-tuning it.
• Per-dataset results: The comparison table (Table 2) is truncated in the supplied content; individual AUROC values per target domain and the scores of DANN, ADDA, MDAN, DECISION, Bi-ATEN, EAAF, and the Source Only baseline are not reported in the text available here.
• Ablation and efficiency analyses: The paper states that ablation studies and efficiency analyses were conducted, and Section 4.6.5 is cited as the source for the temperature choice τ = 0.2; the individual ablation numbers are not included in the provided content.
• Measured domain shift: Pairwise Fréchet Distances between the five datasets' feature distributions are reported as substantial, which the authors use to justify the need for domain adaptation.
• Theoretical support: Under an assumption that each teacher's logit equals a latent logit plus a teacher-specific error, the authors show that averaging teacher logits reduces the mean-squared error magnitude as the number of teachers M grows, provided teacher errors are not perfectly correlated; and that minimizing the KL divergence to the consensus distribution has a unique optimum at the consensus softmax.

Methodology in Plain English

• Setting up the privacy problem: Several hospitals each contribute a labeled skin-lesion dataset, treated as a separate "domain." A model must be adapted to a new, unlabeled target dataset using only the trained models from those sources — not the images themselves. Each leave-one-domain-out run holds out one dataset as the target and uses the rest as sources.
• Stage 1 — shared feature space (CAT): Start from an ImageNet-pretrained ViT-B/16 and freeze it. Add small convolutional bypass modules (ConvPass) alongside the transformer's attention and feed-forward blocks, and train only those. The training signal is supervised contrastive learning: images with the same label (melanoma or nevus) are pulled together in feature space and images with different labels pushed apart, regardless of which dataset they came from. Because both classes and datasets are heavily imbalanced (for example, 1113 melanoma versus 6705 nevus in ISIC 2018), mini-batches are built by weighted random sampling with weights that reduce to N/n_ij — the inverse frequency of each (domain, class) combination.
• Stage 1.5 — source models: Freeze the resulting backbone. For each source domain, attach its own small bottleneck (four fully connected layers of 2048, 1024, 512, and 256 nodes) and a single-layer classification head, and train them with binary cross-entropy.
• Stage 2 — adaptation without source data (CKD): For each unlabeled target image, run all source models and average their logits to form a consensus. Train a student model to match the softmax of that consensus using KL divergence. Soft targets are used instead of hard pseudo-labels so the student matches a full probability distribution rather than committing to a possibly wrong one-hot label, and simple averaging is used instead of confidence weighting so overconfident teachers cannot dominate.
• Training details: Batch size 32 everywhere; source models trained 20 epochs; student trained 15 epochs; SGD with momentum 0.9; learning rate 3×10⁻²; warmup over 5% of total steps; contrastive temperature τ = 0.2. Augmentation is resize, random horizontal and vertical flips, and random rotations between 0 and 90 degrees.
• Evaluation: AUROC, chosen because it is threshold-independent and robust to the severe class imbalance typical of these datasets.

Why This Matters

• For research: The paper argues that the field has been solving domain shift at the aggregation stage while neglecting the feature space itself, and that full backbone fine-tuning is the wrong default for small, imbalanced clinical datasets. It offers a concrete recipe — contrastively trained adapters plus consensus distillation — and a theoretical sketch of why averaging teachers stabilizes supervision, which is testable and reusable beyond dermatology.
• Real-world applications:
  • Deploying a melanoma triage model to a new hospital whose images come from different dermatoscopes, without transferring patient images out of the original institutions.
  • Retrospective validation on legacy archived datasets where the original training data can no longer be shared or even located.
  • Multi-institution research consortia where each site keeps its data local but shares only trained models.
  • Adapting between imaging modalities — the paper explicitly splits Derm7pt into dermoscopic and clinical subsets because their distributions differ.
• Industry relevance: Regulatory and contractual constraints on moving patient data make source-free adaptation commercially attractive for medical AI vendors. The 4.3M trainable parameter count matters practically: adapters and small heads are cheap to store and ship per customer site, and they avoid retraining a full ViT for every deployment.

Future Directions

• Fair head-to-head numbers: The truncated results table makes it hard to see where CAT-CKD wins and loses per target domain; a per-domain breakdown against DECISION, Bi-ATEN, and EAAF, including the Source Only baseline, is the obvious follow-up needed to judge robustness.
• Beyond binary melanoma/nevus: All five datasets were restricted to melanoma and nevus because both classes appear in every dataset. Extending to the full lesion taxonomy across heterogeneous label sets is an open problem the current design sidesteps.
• Stronger consensus rules: The paper deliberately uses plain logit averaging to avoid the failure modes of confidence weighting. Whether calibrated, disagreement-aware, or uncertainty-aware aggregation can beat averaging while keeping the stability guarantee is left open.
• Adapter capacity and backbone choice: Only ViT-B/16 with ConvPass and a fixed four-layer bottleneck (2048/1024/512/256) is studied; how adapter placement, bottleneck width, and larger or domain-specific backbones trade off against the 4.3M-parameter budget is not explored in the provided content.

Target Audience

Researchers and practitioners working on domain adaptation, source-free or federated-style learning, and parameter-efficient fine-tuning, particularly those applying vision transformers to medical imaging under data-sharing restrictions. It is also relevant to clinical-AI engineers who need to ship models to new hospital sites without moving patient data, and to graduate students looking for a compact, well-structured example of combining contrastive adapter training with multi-teacher distillation. Readers without background in domain adaptation, contrastive losses, or knowledge distillation will find the framework's motivation accessible but the method sections demanding.

Authors’ abstract

Skin cancer diagnosis, particularly differentiating melanoma from benign nevi, is challenging due to the visual similarity between lesions. Although convolutional neural networks (CNNs) and vision transformers (ViTs) have shown promising performance, their effectiveness often deteriorates on data from heterogeneous clinical sources. Conventional domain adaptation methods address domain shift but require access to source data during adaptation, which is often infeasible due to privacy regulations. Multi-source-free unsupervised domain adaptation (MSFDA) addresses this limitation by leveraging multiple labeled source domains to generalize to an unlabeled target domain without requiring access to source data. However, existing MSFDA methods rely on full backbone fine-tuning, leading to catastrophic forgetting and overfitting on small clinical datasets, and address domain shift at the aggregation stage without establishing a shared domain-invariant feature space. Furthermore, their reliance on hard pseudo-labels or confidence-weighted aggregation introduces noisy supervision signals under domain shift. To address these limitations, we propose CAT-CKD, consisting of two components: (1) contrastive adapter training (CAT), which trains lightweight ConvPass adapters within a frozen ViT backbone using supervised contrastive learning (SCL) to establish a shared domain-invariant feature space before source-specific model training, and (2) consensus knowledge distillation (CKD), which aggregates logits from multiple source models into a consensus supervisory signal and adapts a student model on unlabeled target data using KL divergence. Experiments on five publicly available skin lesion datasets show that CAT-CKD achieves an average AUROC of 86.1%, outperforming existing MSFDA methods while requiring only 4.3M trainable parameters. The code is available at https://github.com/A-Abedi/CAT_CKD.