DistillPath: An Efficient 22M Distilled Pathology Encoder Approaching Large Foundation Model Performance

AI summary

Overview

Research area: Computational pathology, vision foundation models, and knowledge distillation (computer vision / medical imaging).

Technical level: Intermediate. The paper assumes familiarity with Vision Transformers (ViT), self-supervised pretraining objectives such as DINO and iBOT, and standard knowledge-distillation losses, but the method is described in enough detail for a reader with a machine-learning background to follow it.

Scope (one sentence): The paper distills four released pathology foundation models (86M to 1.1B parameters) into a single 22M ViT-S/16 encoder using only frozen class and patch tokens and 6,000 public TCGA slides, then measures the resulting performance-efficiency tradeoff on the EVA, HEST, and PLISM benchmarks.

What This Paper Is About

Pathology pipelines split whole-slide images into thousands of tiles and encode every tile with a pretrained model, but the strongest tile encoders now have hundreds of millions to more than a billion parameters, which makes whole-slide inference and feature storage expensive on commodity hardware. The authors ask whether a compact 22M encoder can inherit most of the useful signal from these large models without repeating their large-scale pretraining. Their answer is DistillPath-KS16, a family of distilled kaiko ViT-S/16 encoders trained by matching only the frozen teachers' final class and patch tokens on public slides.

Key Contributions

1. DistillPath-KS16 encoder family. An efficient 22M pathology encoder family based on the existing kaiko ViT-S/16, with 384-dimensional outputs, distilled separately from four released teachers: H0-mini (86M), Virchow2 (632M), UNI2-h (681M), and H-optimus-0 (1.1B).

2. A backbone-token-only distillation recipe. The recipe reads only frozen teacher class and patch tokens, uses no teacher DINO or iBOT pretraining heads, and trains on 6,000 public slides, so it applies to any released encoder that exposes backbone tokens. It combines a pointwise cosine loss and a relational RKD loss on the class token with a pointwise cosine loss on the patch tokens.

3. A quantified performance-efficiency tradeoff. The best variant, DistillPath-KS16-Virchow2, reaches 0.795 on the seven-task EVA mean, within 0.015 points of the 0.810 Virchow2 class-token reference at about 29 times fewer parameters, and runs more than 25 times faster than Virchow2 in the authors' encoder-forward benchmark.

4. Evidence that teacher choice is task-dependent. The paper shows that teacher size or teacher EVA score does not predict the student's performance, and that different teachers win on different benchmarks.

Main Findings

• Every distilled variant beats the baseline on all three benchmarks. All four DistillPath-KS16 variants improve on the undistilled kaiko ViT-S/16 baseline: EVA mean rises from 0.764, HEST mean from 0.349, and PLISM from 0.307, while individual tasks can decrease.

• Best EVA result: DistillPath-KS16-Virchow2 reaches 0.795 on the seven-task EVA mean, compared with the 0.810 Virchow2 class-token reference, and scores above H0-mini (0.784) and GPFM (0.789) on this aggregate metric.

• The aggregate EVA advantage is task-concentrated, not uniform. Relative to kaiko, DistillPath-KS16-Virchow2 raises BreakHis from 0.720 to 0.849 (above its own teacher's 0.821, and the only task on which it beats the teacher) and Gleason from 0.723 to 0.774; MHIST drops and MoNuSAC is essentially unchanged. Counting individual tasks, it trails H0-mini on five tasks and GPFM on four.

• HEST favors a different teacher. The best HEST student is DistillPath-KS16-H0mini at 0.387, versus the kaiko baseline of 0.349 and the standalone H0-mini reference of 0.396. DistillPath-KS16-Virchow2 is the weakest HEST student at 0.371 and is not best on any individual HEST task.

• PLISM favors yet another ordering. Distillation improves the kaiko PLISM score from 0.307 to a range of 0.447–0.495. H0-mini remains the strongest reference encoder, and the H0-mini-distilled student is the strongest variant, while the H-optimus-0-distilled student gives the best distilled top-5 retrieval under scanner and staining changes.

• The largest teacher gives the lowest-EVA student. H-optimus-0 is the largest teacher and scores 0.803 EVA, yet DistillPath-KS16-HOpt0 has the lowest EVA mean among the four students at 0.769. UNI2-h, the second strongest teacher at 0.806, gives a student close to H0-mini at 0.772.

• Inference cost: At batch size 64, DistillPath-KS16-Virchow2 reaches 7994 tiles/s on an NVIDIA RTX 4090 (bfloat16) and 396.1 tiles/s on a MacBook Pro M4 Pro (MPS, fp32). It is faster than H0-mini, GPFM, Virchow2, and H-optimus-0 by 4.4x, 13.3x, 26.6x, and 44.2x on the RTX 4090, and by 4.9x, 14.9x, 30.9x, and 53.5x on the MacBook Pro.

• Memory and storage: Relative to Virchow2 at batch size 64, the best variant uses 12.2 times less peak CUDA memory and 19.3 times less measured MPS allocated memory, and requires 3.3 times less fp32 storage for class-token features (1.43 GiB per one million tile embeddings versus 4.77 GiB).

• Training dynamics differ by benchmark. At 10,000 steps H0-mini is strongest on both EVA and HEST; by 20,000 steps Virchow2 is the strongest EVA student at 0.780, while H-optimus-0 is still at the rounded kaiko baseline of 0.764, so 30,000 steps is the first checkpoint at which all four runs exceed baseline. From 30,000 to 50,000 steps each run varies by at most 0.005 EVA. The H0-mini student is strongest on HEST at every checkpoint, improving from 0.381 at 10,000 steps to 0.387 at 50,000 steps, while the Virchow2 student reaches 0.375 HEST at 20,000–30,000 steps and then falls to 0.371.

• The recipe also works with a non-pathology student, but less well. Starting from an ImageNet-21k ViT-S/16 (baseline 0.729 EVA, 0.311 HEST, 0.383 PLISM) rather than kaiko, distillation raises EVA to 0.754–0.768, HEST to 0.358–0.377, and PLISM to 0.490–0.561. H-optimus-0 gives the best ImageNet-initialized EVA student (0.768), H0-mini the best HEST (0.377), and UNI2-h the best PLISM (0.561), which is above both the H0-mini reference and the kaiko-initialized DistillPath variants.

Methodology in Plain English

Student and teachers. The student is the 22M kaiko ViT-S/16, initialized from public pathology-pretrained kaiko weights and producing 384-dimensional features. Teachers are frozen and range from 86M (H0-mini, ViT-B/14) to 1.1B parameters (H-optimus-0, ViT-g/14), with output dimensions of 768, 1280, 1536, and 1536.

What is transferred. The same augmented 224x224 tile is passed to both the teacher and the student. Only the teacher's final class token and final patch tokens are used; the teacher's pretraining heads and register tokens are ignored, and no gradients flow into the teacher.

Losses. On the class token the method combines a pointwise cosine loss (mapping the student feature into the teacher's dimension with a shared DINO-style projector) with a relational RKD loss that matches pairwise distances and triplet angles within a batch, scaled by a teacher-specific coefficient. RKD needs no projector because it compares geometry inside each representation space. On the patch tokens a cosine loss is applied after the teacher's 16x16 grid is resized to the student's 14x14 grid with PyTorch bicubic interpolation; no patch-level RKD is used because the resized patches already have explicit spatial correspondences.

Loss balancing. The authors use one standardized loss-contribution recipe: in the late training regime, RKD contributes about 25% of the class-token loss and patch supervision about 25% of the total objective, measured over steps 40,000–50,000. Teacher-specific coefficients γ and λ range from 24 to 84 and 0.16 to 0.31 respectively.

Training. Each run uses 50,000 steps at batch size 256 in bfloat16, AdamW with learning rate 10⁻⁴, weight decay 0.05, 500 warmup steps, cosine decay to 10⁻⁶, and gradient clipping at norm 3.0, taking about 24–29 GPU-hours on one NVIDIA RTX 4090.

Data. Training uses 6,000 public TCGA H&E whole-slide images from 32 cancer cohorts, with cohort frequencies following the observed TCGA distribution without rebalancing, totaling 12.8 million accepted tile views. Tiles are streamed online from slides rather than pre-extracted, using the authors' wsistream library, at 0.25, 0.5, 1.0, and 2.0 microns per pixel, keeping tiles with at least 40% tissue and filtering low-information tiles in HSV space with HED color augmentation at strength σ = 0.08. The accepted-tile magnification mix was 44.9% at about 0.25 mpp, 7.2% at about 0.5 mpp, 44.8% at about 1.0 mpp, 2.4% at about 2.0 mpp, and 0.7% other or unknown.

Evaluation. All models are evaluated with identical protocols. EVA uses tile-level classification and segmentation tasks (BACH, CRC, PCam, MHIST, BreakHis, Gleason, CoNSeP, MoNuSAC) with balanced accuracy for classification and MonaiDice for CoNSeP and MoNuSAC, averaged over five probe runs; BACH is shown separately and the EVA mean covers the remaining seven tasks. HEST predicts spatial transcriptomics across nine tasks by reducing class tokens to 256 dimensions with PCA and fitting ridge regression, reporting mean Pearson correlation. PLISM measures representation consistency under scanner and staining changes on an 8,139-tile protocol.

Why This Matters

Impact on research. The paper demonstrates that released pathology foundation models can be compressed into a fixed 22M student using only their frozen backbone tokens, without access to their pretraining heads or a billion-tile corpus. It also provides evidence that the strongest or largest teacher is not automatically the best teacher for a given target task, since EVA, HEST, and PLISM rank the students differently.

Real-world applications:

• Deploying whole-slide-image tile encoding on commodity or clinical workstation hardware, where storing embeddings for thousands of tiles per slide is otherwise costly.
• Large-scale retrospective slide archives, where feature storage and inference time dominate the cost of building datasets.
• Cross-scanner and cross-staining retrieval or quality-control pipelines, using the PLISM results on matched tissue under scanner and staining changes.
• Spatial-transcriptomics-adjacent workflows, where HEST evaluates gene-expression prediction from compact image features.

Industry relevance. The cost argument is the central practical claim: the EVA-best student is reported as 26.6 times faster on an RTX 4090, 30.9 times faster on a MacBook Pro M4 Pro, and needs 3.3 times less fp32 feature storage than Virchow2, while returning 384-dimensional features that require less device memory than teachers with 1280- or 1536-dimensional outputs. Both code and released model weights are public.

Future Directions

• Test a larger student. Because the main student has 384-dimensional outputs, the authors state they cannot tell whether weaker transfer from the largest 1536-dimensional teachers reflects teacher-student mismatch, the training recipe, or a genuine capacity bottleneck; a ViT-Base student is named as a natural next test.
• Run a full controlled ablation. The paper uses one standardized loss-contribution balance for the four-teacher comparison and explicitly notes it does not present a controlled ablation across all teachers, coefficients, schedules, and calibrated final settings, and lacks a no-teacher continued-training control.
• Add multiple teachers or a self-supervised objective. Multi-teacher distillation is described as a natural direction that this work does not explore, since it isolates one teacher at a time; the authors also note they do not combine feature matching with a self-supervised objective, unlike GPFM.
• Broaden and rebalance the training data. All training slides come from TCGA, which is less diverse than the collections used to train the teachers; results may depend on TCGA cohort composition, magnification distribution, and tissue filtering, and a more diverse or rebalanced set might improve transfer.
• Compare against the closest prior model under identical conditions. Virchow2G-Mini is described as the closest conceptual comparison (a 22M ViT-S distilled via a DINOv2-style recipe from teacher heads on one billion tiles), but its weights and matching benchmark outputs are not publicly available, so it is not used as an experimental baseline.

Target Audience

Researchers and engineers working on computational pathology, whole-slide-image representation learning, and foundation-model compression, particularly those who need compact tile encoders for large-scale slide processing. It is also relevant to practitioners interested in knowledge distillation from closed or head-less released models, and to benchmark-oriented readers who want a careful account of how performance rankings change across EVA, HEST, and PLISM.

Authors’ abstract

Many high-performing pathology tile encoders are now foundation models with hundreds of millions to over a billion parameters. Encoding and storing the thousands of tiles in each whole-slide image with such models is costly on commodity hardware, so compact encoders that retain useful downstream performance are a valuable alternative. We present DistillPath-KS16, which starts from the existing 22M kaiko ViT-S/16 encoder and improves it by distilling from released pathology encoders used as frozen teachers. The recipe reads only the teachers' final class and patch tokens and trains on 6,000 public slides, needing neither their DINO nor iBOT pretraining heads nor a billion-tile corpus, so it applies to any released encoder that exposes backbone tokens. We distill four teachers spanning 86M to 1.1B parameters into the same student. Every variant improves the kaiko baseline on all three benchmarks we use, EVA, HEST, and PLISM, and the strongest teacher is task-dependent. On the seven-task EVA mean, DistillPath-KS16-Virchow2 reaches $0.795$, within $0.015$ points of Virchow2, the top-scoring model in our evaluation, at about $29\times$ fewer parameters; it also scores above H0-mini and GPFM on this aggregate metric, though that advantage is task-concentrated rather than uniform. Because it remains a 22M ViT-S/16 with 384-dimensional features, DistillPath-KS16 runs more than $25\times$ faster than Virchow2. Code is available at https://github.com/RamonKaspar/DistillPath, and released model weights are available at https://huggingface.co/collections/RamonK/distillpath.