Beyond Sparse Captions: Aligning Slide-Level Text and Patch-Level Vision in Pathology

AI summary

Overview

Research area: Computational pathology; contrastive vision-language pretraining; multiple instance learning (MIL) on whole-slide images.

Technical level: Advanced. The paper assumes familiarity with contrastive objectives (InfoNCE), cross-attention pooling, and MIL aggregation over patch bags.

Scope: This paper introduces LIMIT, a locked-image text-tuning recipe that aligns a pathology text encoder to slide-level clinical reports, producing the CONCH-Z model, evaluated across 22 zero-shot tasks on five datasets.

What This Paper Is About

Pathology vision-language models are currently trained on image-caption corpora scraped from social media, educational videos, and publications — datasets typically on the order of one million curated pairs that are sparse relative to tissue morphology diversity and hard to scale, because pathologists do not routinely write descriptive captions. The paper instead proposes using slide-level pathology reports generated during routine diagnostic workflows as supervision, and solves the resulting mismatch: reports describe macroscopic diagnostic findings while patches describe microscopic morphology. LIMIT bridges this gap by freezing a pretrained vision encoder and fine-tuning only the text encoder, using report embeddings as cross-attention queries over bags of patch features.

Key Contributions

1. The authors formalize vision-language pretraining for pathology as a weakly-supervised cross-modal MIL problem and introduce LIMIT, which fine-tunes only the text encoder via report-guided cross-attention over frozen patch embeddings.
2. They train CONCH-Z on 335,645 clinical reports and report state-of-the-art zero-shot performance across diverse benchmarks, including pan-cancer subtyping, tumor detection, and patch-level classification.
3. They show that slide-level report supervision, mediated through text-guided attention, strengthens fine-grained visual-semantic alignment, yielding simultaneous improvements at both patch and slide level without local annotations or learned aggregation parameters.
4. They release code and model at https://github.com/mahmoodlab/LIMIT.

Main Findings

• Patch-level zero-shot (SPIDER): CONCH-Z reaches overall balanced accuracy of 58.8% on SPIDER-C versus MUSK 51.3%, CONCH 49.7%, Mi-Zero 44.2%, PLIP 34.8%, and QuiltNet 30.1%. It leads on Colorectal (54.8%), Breast (70.2%), and Thorax (49.4%), and is second to MUSK on Skin (60.8%). On fine-grained SPIDER-F it leads all organs, with an overall 51.3%.
• Out-of-domain canine tissue (CATCH): CONCH-Z reports the best overall balance accuracy of 64.4%, with 83.7% on coarse cancer detection (Skin-C) and 45.1% on fine-grained subtyping (Skin-F).
• Tumor detection (CAMELYON-16): A zero-shot AUC of 96.7%, above QuiltNet (91.7%) and CONCHv1.5 (90.3%). Prompting with diagnostic terms produces overlap with pathologist-annotated regions of metastasis, adipose tissue, and lymphocytes, giving inherent spatial interpretability.
• Slide-level subtyping (TCGA-OT): 78.8% overall balanced accuracy, an absolute +21.4% over its base vision backbone CONCH (57.4%) and +5.7% over TITAN (73.1%); PRISM is reported at 59.4%.
• Slide-level subtyping (SlideChat-HistAI): 54.6% overall, above the next-best patch model MUSK (44.6%) and slide model PRISM (50.2%); TITAN is reported at 46.1%.
• Versus supervised MIL on TCGA-OT: Fully supervised ABMIL reaches 73.4%, TransMIL 71.0%, DSMIL 71.1%, CLAM 71.5%, and WiKG 70.9%. A fixed "cancer"-query linear probe on CONCH-Z features (CONCH-Z_lin) reaches 76.0%, above ABMIL by +2.6%, while end-to-end zero-shot CONCH-Z reaches 80.0%, which is +6.6% over ABMIL and +4.0% over CONCH-Z_lin.
• Pretraining scale: Balanced accuracy averaged over TCGA-OT and HistAI rises consistently with data volume, from 62.7% at 50k to 66.6% at 340k (the full 335,645 samples).
• Negative sampling: Sampling negatives from random organs instead of the same organ causes a 2.9% relative drop at 340k and 3.5% at 50k (1.9 and 2.2 percentage points). Using 1 or 2 negatives degrades performance by up to 7.1% relatively at 340k.
• Loss weighting: Removing the similarity-based weighted loss causes an 8.1 percentage point drop (12.2% relative) at 340k. Replacing InfoNCE with weighted SigLIT loss tracks closely (66.3% versus 66.6% at 340k).
• Parameter-efficient tuning: LoRA fine-tuning underperforms full text encoder fine-tuning across all scales (64.1% versus 66.6% at 340k).
• Pooling: Mean pooling drops performance 3–8% relative and random pooling about 11% relative versus LIMIT weighted pooling. Top-10 and top-100 patch aggregation are comparable but vary across training scales.
• Prompt robustness: Substituting five cancer-related synonyms across three sentence templates keeps CONCH-Z above existing vision-language baselines, including the frozen CONCHv1.5 backbone.

Methodology in Plain English

Each whole-slide image is cut into non-overlapping patches at 448×448 pixels by default, and a frozen, pretrained vision encoder turns them into patch embeddings. The text encoder — the only trainable part — embeds a candidate pathology report into a vector, and that vector is used as a query: the model computes dot-product similarity between the report embedding and every patch embedding, softmax-normalizes those scores with a temperature, and takes a weighted sum of the patches. The result is a slide representation that is conditioned on whichever report it is being compared against.

Training pairs each slide with its true report as a positive, plus four additional reports drawn exclusively from the same organ type as hard negatives. Because same-organ reports often describe identical cancer subtypes and grades, the authors compute the cosine similarity between the positive and each negative report and use it to downweight the penalty applied to semantically overlapping negatives, with gradients blocked from the weighting term. The objective is a similarity-weighted InfoNCE loss with temperature 0.07, averaged over all slides in the bag. Gradients flow only into the text encoder.

The model was trained using pre-computed patch features from the Trident library, for 3 epochs with batch size 1, learning rate 1×10⁻⁵, weight decay 1×10⁻⁵, AdamW, and a cosine annealing schedule, holding out a 1% validation split for epoch selection. Reports were processed with a locally hosted Qwen3-14B model to remove identifying information and filter non-histology content. At inference, the model encodes K candidate texts, aggregates the slide conditioned on each candidate, and picks the candidate with maximum cosine similarity — a retrieval formulation requiring no linear probing or downstream fine-tuning. Zero-shot labels follow the CPath convention of prepending tissue type to the class label. For comparison, supervised MIL baselines were trained with AdamW at learning rate 1×10⁻⁴, cross-entropy loss, random class-weighted sampling, batch size 1, max 20 epochs with early stopping (5-epoch patience, 10-epoch minimum), dropout of 0.25 in feedforward layers and 0.1 on frozen-encoder features, on three NVIDIA RTX 3090 GPUs, with 3 epochs on the full training set taking 31 hours.

Training slides came from the digital archives of Massachusetts General Hospital and Brigham and Women's Hospital under IRB approval, spanning 335,645 H&E and IHC stained WSIs across 20 organs, the same slides used in TITAN. No TCGA, CAMELYON, HistAI/SPIDER, or CATCH slides or reports were included in training.

Why This Matters

Impact on research: The work reframes pathology vision-language pretraining away from curated captions toward clinical reports already produced during routine sign-out, which are orders of magnitude more plentiful in hospital archives. It also demonstrates that text-only adaptation can beat highly parameterized slide-level foundation models on several zero-shot benchmarks without training a new slide encoder or learned aggregator.

Real-world applications:

• Zero-shot tissue and subtype classification at patch and slide level without task-specific training data or labels.
• Zero-shot metastasis detection with prompt-driven spatial heatmaps that overlap pathologist annotations, usable for triage and quality control.
• Cross-species and cross-domain generalization, demonstrated on canine cutaneous biopsies (CATCH).
• Prompt-driven organ-specific multiple-choice subtyping that mirrors clinical workflows where tissue origin is known in advance.

Industry relevance: The frozen-image design carries no tunable image-side parameters, which keeps training cost low and preserves the patch-level visual space already used by downstream slide architectures, explainability frameworks, and reasoning systems. That makes the text encoder a drop-in upgrade rather than a new backbone, and it reduces the risk of degrading learned morphological features through noisy report supervision.

Future Directions

• Extending evaluation beyond diagnosis, subtyping, and tissue classification to global slide-level properties such as grade, tumor cellularity, extent of necrosis, or global mitotic activity, which the authors state may need modeling of relationships across many regions.
• Exploring lightweight image-side adaptation while retaining the benefits of locked-image tuning, since the current design may be less expressive for holistic slide-level properties.
• Better understanding when top-k patch aggregation helps versus weighted aggregation of all patches, which varied across training scales and depends on choosing k appropriately.
• Characterizing the scaling curve beyond 335,645 reports, since performance was still rising with data volume at the largest pretraining size tested.

Target Audience

Researchers and engineers working on computational pathology foundation models, multimodal retrieval, and multiple instance learning; clinical AI teams seeking scalable supervision from routine diagnostic reports; and machine learning practitioners interested in locked-image tuning, cross-attention pooling, and hard-negative weighting in weakly supervised contrastive learning.

Authors’ abstract

Contrastive vision-language (VL) pretraining in computational pathology has generally relied on curated image-caption datasets sourced from social media, educational videos, and research publications. These corpora are sparse relative to the diversity of tissue morphology and difficult to scale further, as pathologists do not routinely produce descriptive captions in clinical practice. We propose to instead leverage slide-level pathology reports which can be generated during routine diagnostic workflows in a scalable manner. A key challenge in this weakly-supervised setting is that pathology reports describe macroscopic, diagnostic findings while patch-level visual features describe microscopic findings. We introduce Locked Image Multiple Instance PreTraining (LIMIT), which addresses this gap by freezing a pretrained vision encoder and fine-tuning only the text encoder: report embeddings serve as cross-attention queries over bags of patch features, producing report-conditioned slide representations optimized via InfoNCE. Using LIMIT, we establish CONCH-Z, which aligns the text encoder of CONCH v1.5 to 335k clinical reports. Evaluated across 22 tasks spanning patch-level classification, tumor detection, and slide-level subtyping, CONCH-Z establishes state-of-the-art zero-shot performance over VL encoders trained on curated caption datasets, while simultaneously closing the gap with slide-level foundation models trained with substantially more parameters and compute. The code and model is available at https://github.com/mahmoodlab/LIMIT.