PRISM: Pathology Reliability In Scarce-label Medicine. A Tile-level Benchmark of Pathology Foundation Models

AI summary

Overview

Research area: Computational pathology / medical machine learning — specifically the evaluation of pathology foundation models (large self-supervised and vision-language models trained on whole-slide images) under realistic clinical constraints.

Technical level: Advanced. The paper assumes fluency with AUROC, Expected Calibration Error, temperature scaling, linear probing on frozen embeddings, meta-analysis statistics (Spearman ρ, I²), and domain-shift terminology.

Scope (one sentence): PRISM is a tile-level benchmark that jointly measures eight pathology foundation models along four reliability axes — model choice, label fraction (1% to 100%), calibration, and out-of-distribution transfer — across six datasets and a five-hospital Camelyon17 transfer grid, totaling 864 in-distribution linear probe runs and 3,600 transfer runs.

What This Paper Is About

Pathology foundation models are almost always judged by discrimination (AUROC or accuracy) at full label availability on in-distribution test sets, which says nothing about whether a model can express appropriate uncertainty when it should. PRISM asks whether a model's ranking on discrimination still predicts its ranking on calibration when labels are scarce, and whether additional source-domain labels actually help or harm a probe's calibration once it is transferred to a new institution or a new label definition.

Key Contributions

1. A unified four-axis evaluation. Eight foundation models (CLIP, PLIP, CONCH, UNI, VIRCHOW2, GigaPath, H-Optimus-0, MIDNIGHT) are evaluated on six tile-level datasets at six label fractions with three seeds — 864 linear probe runs — extended with a five-hospital Camelyon17 grid of 20 directed transfer pairs (3,600 further runs). The authors state that no prior study evaluates the joint interaction of model choice, label fraction, OOD shift, and calibration.

2. The Composite Reliability Index (CRI). A multiplicative index, CRI = AUROC × (1 − scaled ECE) × OOD_Stability, deliberately built so a model scoring zero on any one axis cannot achieve a high composite. The authors explicitly present it as a screening heuristic reported alongside its three components, not as a replacement for them.

3. A transfer-quality explanation for reverse calibration scaling. The paper shows that whether extra source labels help or hurt target calibration tracks transfer AUROC rather than the mere presence of shift, with a sign change near transfer AUROC 0.85 in the evaluated settings.

4. An open, pip-installable package (prism_bench). The package ships a PRISMEvaluator API and reference results for all 288 (model, dataset, label fraction) cells, pre-computed embeddings where licences permit redistribution (five of six datasets; not MHIST), and extraction scripts with SHA256 checksums for MHIST, UNI, CONCH, and VIRCHOW2.

Main Findings

• Discrimination and calibration rankings decouple as labels shrink. Pooled within-dataset Spearman correlation between AUROC rank and post-temperature ECE rank is ρ = 0.52 at full supervision (I² = 0% between datasets) but ρ = −0.24 at 1% labels (I² = 67%), a difference of Δρ = 0.62 (95% CI [0.08, 1.13]). On raw ECE, which uses no validation labels, the contrast is larger: ρ = −0.46 [−0.85, −0.05] at 1% versus 0.47 [0.03, 0.86] at full supervision, Δρ = 0.93 [0.51, 1.41].

• The decoupling runs in both directions. A model's calibration rank sits a mean of 3.0 of 8 positions from its AUROC rank at 1% labels versus 1.8 at full supervision. CLIP ranks last on AUROC yet first on post-temperature ECE on BRACS, LungHist700, and PCam — on BRACS its probe assigns 92% of the test set to one class (macro-F1 0.08), so it is well calibrated largely because it commits to almost nothing. Conversely, H-Optimus-0 has the top AUROC on PCam at 1% labels (0.982) but ranks seventh of eight on calibration.

• Linear probes collapse to a single predicted class on harder datasets. On MHIST every model assigns more than 99% of the test set to one class at 1% labels; on PCam and CRC this never occurs, and stratified sampling does not remove the collapse. CLIP and PLIP never reach macro-F1 = 0.5 on BRACS, while UNI and VIRCHOW2 cross it on every dataset by 25% labels and GigaPath and H-Optimus-0 by 50%. At 10% labels only four models (CONCH, GigaPath, UNI, VIRCHOW2) reach the threshold on BRACS and none on MHIST.

• Additional source labels can worsen target calibration. On the four cross-domain pairs, more source labels increase target ECE. On MHIST→PCam the four large vision-only SSL models rise monotonically across seeds: UNI from 0.225 to 0.439, VIRCHOW2 from 0.231 to 0.453, GigaPath from 0.243 to 0.462, and H-Optimus-0 from 0.232 to 0.488. At 100% labels GigaPath reaches OOD AUROC 0.487 with OOD ECE 0.462, against an in-distribution ECE of 0.042.

• The effect reverses under covariate shift alone. On the 20 Camelyon17 hospital pairs, where only scanner, staining, and patient population differ, target ECE rises in only 10 of 160 (model, pair) combinations rather than 28 of 32, with a mean change of −0.149 rather than +0.082. For UNI, VIRCHOW2, GigaPath, and H-Optimus-0, ECE rises in 1 of 80 combinations with a mean change of −0.184.

• Transfer quality, not shift type, predicts the sign. Across all 192 (model, pair) combinations the Spearman correlation between OOD AUROC and the ECE slope is −0.78 [−0.91, −0.59], and within Camelyon17 alone it is −0.65 [−0.87, −0.18]. Binned by transfer AUROC, ECE rises in 88%, 88%, and 86% of combinations below 0.55, between 0.55 and 0.70, and between 0.70 and 0.85, then in 8% between 0.85 and 0.95 and 1% above 0.95 — an 86-percentage-point difference [66, 100].

• Degenerate prediction does not explain it. Single-class collapse occurs in 57% of label-definition cells but only 0.7% of hospital cells. Restricting to the 163 of 192 combinations that never collapse, transfer AUROC still predicts the slope at −0.66 [−0.87, −0.27]; in a joint regression, transfer AUROC accounts for R² = 0.65 against 0.44 for degeneracy alone.

• Miscalibration is predominantly under-confidence. Fitted temperatures indicate T < 1 in 189 of 192 multiclass cells (T between 0.28 and 0.40 on SPIDER-Breast at 1% labels) and in 63 of 96 binary cells — 252 of 288 cells overall. Over-confidence appears mainly on PCam from 25% labels and on MHIST at 1–5%, where the probe is collapsed.

• Temperature scaling is not uniformly effective. It removes more than 0.4 of ECE in some cells (H-Optimus-0 on SPIDER-Breast at 1% labels: 0.465 to 0.020) but fails to reduce ECE in 49 of 288 cells, concentrated on MHIST, PCam, and LungHist700. A source-fitted temperature is actively harmful when transfer fails: it worsens target ECE in 61% of cells across the four label-definition pairs but only 16% on the hospital pairs.

• The composite index behaves differently across the two designs. VIRCHOW2 leads CRI in both regimes (0.642 at full supervision, 0.564 at 1% labels), with best-to-worst spreads of 0.17 and 0.15. UNI ranks last at full supervision despite in-distribution AUROC 0.954, because its transfer AUROC of 0.49 gives it the lowest OOD_Stability (0.52); CLIP, last on in-distribution AUROC on five of six datasets, places fourth and second because a ratio term rewards a model with little to lose. On Camelyon17, where transfer is meaningful, CRI places the pathology-specific models above CONCH, PLIP, and CLIP and agrees with all three components (Kendall τ = 0.71, 0.93, and 0.86).

Methodology in Plain English

The authors freeze each foundation model and extract one set of image embeddings per dataset, so the models themselves are never fine-tuned. On top of those frozen features they train a simple, fixed-regularization logistic regression "probe" (L2-regularized, C = 1.0) at six training-set sizes — 1%, 5%, 10%, 25%, 50%, and 100% of labels — repeating each with three random seeds. Test sets are held fixed and the splits are identical across models, so differences between models at a given fraction come from the embeddings, not from the split.

For each probe they record AUROC, macro-F1, Brier score, and Expected Calibration Error (15 equal-width bins) both before and after temperature scaling, where the temperature is fitted on a held-out validation split. Because that validation split is held at full size, at 1% labels it supplies 90–96% of all labels a cell consumes — so the authors report raw ECE (which uses no validation labels) alongside scaled ECE as a check.

To study transfer they use two designs. The first is four cross-institution pairs (PCam↔MHIST, CRC↔BRACS) in which tissue, site, and the definition of the positive class all change at once. The second is Camelyon17, released by WILDS: patches from five hospitals annotated for one binary task under a single label definition, sampled at 20,000 patches per hospital, label-stratified 50/50 and split 70/15/15 at patch level. Treating each hospital as a domain gives 20 directed transfer pairs. Because only acquisition conditions differ there, comparing the two designs separates label-definition shift from covariate shift. A two-way cluster bootstrap resampling models and hospitals supplies confidence intervals.

Why This Matters

Impact on research. The paper argues that the dominant evaluation paradigm — AUROC at full labels on an in-distribution split — does not satisfy the FDA Good Machine Learning Practice principles or EU AI Act Article 13, both of which call for calibrated probabilities and documented limitations under distribution shift. It also shows the problem is not marginal: a practitioner selecting on AUROC alone would not select the best-calibrated model, and selecting on ECE alone would favour a near-degenerate one. The finding that calibration degrading as source labels accumulate is a cue to measure transfer before collecting more labels is a concrete, actionable diagnostic.

Real-world applications:

• Model selection for clinical deployment, where the composite index and its components give a down-selection screen across discrimination, calibration, and transfer stability rather than a single accuracy number.
• Label-budget planning in hospitals, where collecting annotations is expensive and the paper shows more source labels can actively worsen calibration when a probe does not transfer.
• Cross-institutional validation, since the five-hospital Camelyon17 grid isolates scanner, staining, and population effects and shows the target hospital matters more than the source for calibration (standard deviation of per-hospital mean raw ECE 0.043 across targets versus 0.020 across sources).
• Post-hoc calibration triage, since a single scalar temperature removes large amounts of error cheaply in most cells but is harmful in a minority — a deployment team can know in advance when to trust it.

Industry relevance. Embedding extraction costs roughly 15 GPU-hours on A100 hardware once, after which all 864 probe runs complete in under 30 minutes on a single CPU and bootstrap intervals take 26 seconds. That amortized cost structure makes it practical for a lab or vendor to position a new model against eight reference models without re-running extraction.

Future Directions

• Validation-tuned regularization. The probe's regularization is fixed at C = 1.0. Across a five-value sweep the AUROC ranking of models is stable (mean Kendall τ = 0.82) but the raw-ECE ranking is not (τ = 0.02), and the 1%-versus-100% decoupling contrast is pronounced at C = 1 (Δρ = 0.93), weaker at C = 10 and 100 (0.06 and 0.37), and reversed at C ≤ 0.1. A replication with validation-tuned C is explicitly left to future work.
• Slide-level evaluation. All findings concern tile-level classification with frozen representations; slide-level evaluation, which the authors note is confounded with the aggregator, is out of scope.
• Testing whether the 0.85 transition generalizes. The authors state the transition point is an association observed on two transfer designs, specific to this probe, these datasets, and these pairs — not a general threshold — and note the within-Camelyon17 rate below the transition rests on CLIP alone (mean AUROC 0.849, rising ECE in 8 of its 20 pairs, while the other seven models transfer at 0.966 to 0.995 and show it in 2 of 140).
• Isolating preprocessing effects. CLIP, PLIP, and MIDNIGHT received ImageNet statistics rather than their documented normalization, which confounds between-model comparisons at a fixed fraction. A slide-disjoint Camelyon17 sensitivity analysis is also left for future work.

Target Audience

Machine learning researchers and benchmark designers working on foundation models for computational pathology; regulatory and clinical-translation scientists who need calibrated probabilities and documented behavior under distribution shift rather than headline AUROC; and hospital or vendor ML engineers who must decide which frozen encoder to deploy and how many labels to collect before stopping. Readers without a statistics background will find the AUROC, ECE, and meta-analysis machinery dense, but the plain-language conclusions — more labels are not always better, and calibration rankings are not discrimination rankings — are accessible without it.

Authors’ abstract

Pathology foundation models are typically benchmarked on classification performance at full label availability and on in-distribution test sets, while calibration, behavior at low label budgets, and robustness under cross-institutional shift are seldom evaluated jointly. We introduce PRISM, a tile-level benchmark that measures the reliability of pathology foundation models along four axes: model choice, label fraction, calibration, and out-of-distribution transfer. PRISM evaluates eight foundation models (CLIP, PLIP, CONCH, UNI, Virchow2, GigaPath, H-Optimus-0, Midnight) on six tile-level datasets at six label fractions (1% to 100%) with three seeds (864 linear probe runs), extended with a five-hospital Camelyon17 study of 20 directed transfer pairs. Four phenomena emerge. (i) Under full supervision, AUROC rank predicts post-temperature calibration rank consistently across datasets (Spearman rho = 0.52, I^2 = 0%), whereas at 1% labels the relationship becomes dataset-specific (rho = -0.24, I^2 = 67%). (ii) On harder datasets the linear probe collapses to a single predicted class below a dataset-specific label fraction. (iii) Whether additional source-domain labels help or harm target calibration tracks how well the probe transfers: below a transfer AUROC of roughly 0.85, more labels worsen target ECE in 86-88% of combinations. (iv) Temperature scaling fails to reduce ECE in 49 of 288 cells, miscalibration is predominantly under-confidence, and a source-fitted temperature is harmful under failing transfer. PRISM is released as a pip-installable package with reference results for all 288 cells. Code and data: https://github.com/alicemkyn/prism-benchmark