PathJev: Decision-Native Vision-Language Modelling for Histopathology

AI summary

Overview

Research area: Computational pathology / medical computer vision, specifically vision–language models that output calibrated probability distributions over run-time candidate answers rather than free text.

Technical level: Intermediate. The core ideas are accessible, but the paper assumes familiarity with fine-tuning (LoRA), calibration metrics (ECE, NLL, Brier), foundation-model features and whole-slide image analysis.

Scope (one sentence): The paper builds and evaluates PathJev-P, a "decision-native" histopathology model that returns a normalised distribution over user-supplied candidate answers in a single forward pass, benchmarking it against a matched generative baseline, zero-shot pathology models, a general medical decision model and a supervised linear probe on contamination-aware test sets.

What This Paper Is About

Pathology vision–language models usually end in a decision (which tissue is this, is carcinoma present), but they typically return similarity scores with an arbitrary scale or a token probability from free text, neither of which is a trustworthy distribution over the candidates a clinician cares about — a problem reported to be especially severe in pathology. The authors ask whether a "decision-native" design, in which the question and a candidate set (including a "none of these" option) are written into the prompt and a small decision head returns a normalised distribution in one forward pass, transfers to histology and generalises to organs never seen in training.

The paper also asks a sharper methodological question: are the claimed calibration gains of decision-native models over generative models a property of the decision head, or an artefact of how the generative model's probabilities are reconstructed for the comparison?

Key Contributions

1. PathJev / PathJev-P, a decision-native model for histopathology. Given an image, a question and a candidate set chosen at run time, it returns a normalised distribution over the candidates (including "none of these") in one forward pass, across three question types: Choice (one of K classes), Noul (yes/no) and Score (ordered levels). It is built on MedGemma-1.5-4B with LoRA adapters and a shared decision head.

2. A deterministic construction of decision data from pathology annotations. Expert labels from three public collections are turned into typed questions with organ-specific candidate sets and a "none of these" option — no generated labels — yielding 40,000 patch states, with splits by patient or slide and whole organs held out from training.

3. Cluster-level labelling of whole slides. Tiles are embedded with a pathology foundation model (H-optimus-0) and grouped by a Gaussian mixture model with soft memberships; PathJev labels each cluster from its most typical tiles, abstains on ambiguous clusters, and propagates cluster probabilities to every tile to form probability maps.

4. An evaluation protocol for decision-native models. It compares against a generative model trained on the same data under several read-outs of its probabilities, candidate orders and image-free inputs, on test sets without documented overlap with the backbone's training data, with an analysis plan fixed and hashed before test-set evaluation.

Main Findings

• Accuracy advantage on clean test sets: PathJev-P reaches balanced accuracy 0.541 (SD 0.015 over 3 seeds) versus 0.492 for the generative baseline, 0.345 for CONCH, 0.288 for the Inner-Jev read-out and 0.213 for the released OmniMed-Jev. Paired differences: +0.049 [+0.038, +0.063] against the generative baseline and +0.196 [+0.161, +0.218] against CONCH, favouring PathJev-P in each of the three seeds.

• The accuracy gain comes from unseen organs: on HepatoBench (liver) and HiESD (stomach), organs absent from training, PathJev-P reaches 0.390 versus 0.277 for the generative baseline and 0.328 for CONCH. On in-domain sets the two trained methods are close (0.692 versus 0.707), and on the seen-source SPIDER sets the generative baseline is slightly ahead (0.810 versus 0.781).

• The calibration advantage depends on the read-out (a key, deliberately hedged result): with the summed candidate log-likelihood read-out fixed in the analysis plan, PathJev-P has the lower ECE (0.204 versus 0.279; difference -0.075 [-0.088, -0.064]). Under a length-normalised read-out of the same generative model, examined post hoc, the two are similar on the clean average (0.201 versus 0.204), the generative model has a lower NLL (1.508 versus 1.586), and it is better calibrated on the unseen organs (ECE 0.110 versus 0.301), where PathJev-P remains over-confident.

• A supervised linear probe is better calibrated in every setting: on H-optimus-0 features it reaches ECE 0.047 on the clean average (balanced accuracy 0.724), 0.021 on the unseen organs (0.772, fitted with labels from the test domain), 0.074 in domain (0.677) and 0.013 on the seen-source sets (0.905).

• OmniMed-Jev's failure is not about input size: the released checkpoint often answers "other"; removing that candidate raises its balanced accuracy to 0.311, and pre-resizing images to 896 instead of 448 pixels yields 0.207.

• Robustness checks: over 10 random candidate orders, PathJev-P's clean balanced accuracy stays between 0.542 and 0.549 although 7.3% of individual predictions change (OmniMed-Jev changes 18.9%). Replacing every image with uniform grey drops both trained methods to chance, showing the question text alone does not separate classes. Removing HepatoBench (whose images cannot be grouped by slide) keeps the sign of the comparison: +0.033 [+0.018, +0.052].

• Temperature scaling helps everything but does not fix the unseen-organ problem: one temperature fitted on the validation mix lowers PathJev-P's clean ECE from 0.204 to 0.154 (generative baseline 0.279 to 0.211), and after scaling PathJev-P has the best language-model NLL on clean sets (1.399 versus 1.597 for CONCH and 1.795 for the generative baseline). On unseen organs, however, scaled CONCH reaches a lower NLL (1.769 versus 2.079). Post hoc, a temperature fitted on one unseen organ and applied to the other lowers PathJev-P's unseen-organ NLL from 2.079 to 1.781, close to the 1.713 oracle fitted on the evaluated set itself.

• Whole-slide demonstration: on four HiESD slides, fine-grained gastric categories are mostly not recognised and most clusters abstain, but the adenocarcinoma-presence question separates carcinoma from non-carcinoma tiles. On the two slides containing carcinoma, tile-level AUROC is 0.983 and 0.942, and cluster-level AUROC 0.946 and 0.928; on the two slides without carcinoma, cluster specificity is 0.970 and 0.987. The authors state these metrics were added after inspecting the first slide and support no claim about diagnostic sensitivity.

Methodology in Plain English

The authors reformulate classification as answering a question with a menu of options. For each histology patch, the expert label is converted mechanically into one or more typed questions — pick one of K tissue classes, yes/no for something like tumour presence, or pick an ordered level such as a Gleason pattern — and the candidate list is written into the prompt at run time rather than baked into a fixed classifier. A "none of these" option is always available.

The backbone is MedGemma-1.5-4B loaded in 4-bit precision. The image is encoded by MedSigLIP and passed into the Gemma-3 decoder, followed by the question and each candidate, with a marker token after each candidate and after the question. A single small decision head reads the hidden states at those markers and produces one score per candidate by summing three terms: a per-candidate linear score, a question–candidate interaction, and a set-attention term through which the candidates of a question attend to each other. A softmax over those scores gives the distribution, so no projection onto the language-model vocabulary is needed. Training is two-stage: first the head alone for 200 optimiser steps with the backbone frozen, then the head plus LoRA adapters (rank 16, alpha 16, dropout 0.05) on images resized to 896×896, with Choice candidates shuffled every time and Score candidates kept in order.

The comparison is deliberately careful. A generative model is fine-tuned with the same backbone, the same LoRA configuration and the same training states, but generates answer text; its candidate probability is a softmax over summed token log-likelihoods, requiring one forward pass per candidate. Other comparators are zero-shot CONCH, the released OmniMed-Jev checkpoint, an Inner-Jev read-out that reads letter-token probabilities from the base model, and a supervised linear probe on H-optimus-0 features. Datasets are assigned roles — train, clean test, seen-source test — based on whether public documentation shows overlap with the backbone's training data, so that liver and stomach serve as genuinely unseen organs. Test sets are capped at 5,000 class-balanced states per set by a fixed draw shared by all methods, metrics use a paired cluster bootstrap that resamples patients or slides, and hypotheses, endpoints and tests were fixed and hashed in an analysis plan before any test-set evaluation, with later analyses labelled post hoc.

Why This Matters

Impact on research. The paper's most distinctive move is methodological honesty about a comparison that flatters the proposed method. By showing that the calibration advantage over a generative baseline reverses under a different, equally defensible read-out of that baseline's probabilities, it sets a precedent that calibration claims for decision-native models should be reported under more than one read-out. The contamination-aware dataset role assignment and the pre-registered analysis plan are also templates that the medical vision–language literature could adopt more widely.

Real-world applications (as motivated or demonstrated in the paper):

• Naming unsupervised tissue clusters in whole-slide images, which the paper notes is currently still done by hand or with uncalibrated keyword scores — with abstention when the model is unsure.
• Screening questions over slides, such as "is carcinoma present", where the cluster-level pipeline separated carcinoma from other tissue while running the model on a fraction of the tiles (471 and 480 representative tiles instead of every tile on two slides).
• Any downstream rule that needs a probability over a user-specified candidate set plus an explicit "none of these" option, rather than free text or an arbitrary similarity score.
• Deploying a decision model into a new organ or laboratory, where the paper's temperature-transfer observation suggests labelled data from one new organ may cheaply correct over-confidence on another.

Industry relevance. The single-forward-pass answer (versus one pass per candidate for the generative read-out) is a concrete inference-cost argument for decision heads in slide-scale processing, though the authors note they did not measure end-to-end time because every tile is still embedded with H-optimus-0. The paper also gives a clear practical recommendation: where labelled data for a fixed question exist, a linear probe on a pathology foundation model is the better tool, being better calibrated in every setting and far more accurate with target-domain labels.

Future Directions

• Training on mosaics of cluster tiles and validating the whole-slide pipeline systematically against pathologists — the authors list both as the next steps, noting that four slides support no claim about diagnostic sensitivity or abstention.
• Testing the temperature-transfer remedy on more organs. The observation that a temperature fitted on one unseen organ corrects most of the over-confidence on the other is explicitly described as "an observation, not a transfer property" with only two organs.
• Establishing whether the gain comes from the decision head itself. The two trained systems differ in objective, number of supervised targets and optimiser steps, so the accuracy gain cannot be attributed to the head alone; the authors offer an untested explanation and leave the ablation open.
• Improving fine-grained accuracy on new organs, which at 0.390 balanced accuracy the authors state "is not yet sufficient", and extending the protocol beyond two unseen organs, one 4-bit backbone and one wording per candidate.

Target Audience

Researchers and practitioners working on medical vision–language models, computational pathology and model calibration; machine-learning engineers building decision-support systems that need probability outputs rather than generated text; and anyone evaluating foundation models under contamination concerns. Readers primarily interested in clinical deployment should note the paper's own caution that its evidence is patch-level plus a four-slide demonstration, not a validated diagnostic tool.

Authors’ abstract

Vision-language models used for decisions in histopathology rarely return calibrated probabilities over the candidates a user cares about. Decision-native models write a question and a candidate set chosen at run time into the input and return a normalised distribution from a small decision head in one forward pass. We build PathJev-P, a decision-native model for histopathology: MedGemma-1.5-4B with LoRA adapters and a decision head, trained on 40,000 patch states from three public collections whose expert labels are turned into typed questions with organ-specific candidate sets and a "none of these" option. In a comparison over 3 seeds with an analysis plan fixed before testing, on test sets without documented overlap with the training data of the backbone, PathJev-P is more accurate than a generative model trained with the same backbone, data and selection rule, the zero-shot pathology model CONCH and a general medical decision model (balanced accuracy 0.541 against 0.492, 0.345 and 0.213); its advantage over the generative model comes from two organs absent from training. The calibration advantage depends on how the generative model's probabilities are read out. With the summed candidate log-likelihood used in previous work, PathJev-P has the lower calibration error (ECE 0.204 against 0.279); with a length-normalised read-out of the same generative model, examined post hoc, the two have similar mean calibration error (0.201) and the generative model is better calibrated on the unseen organs, where PathJev-P remains over-confident. A supervised linear probe on a pathology foundation model is better calibrated in every setting. In a qualitative whole-slide demonstration, PathJev-P labels unsupervised tissue clusters with abstention and separates carcinoma from other tissue while running the model on a fraction of the tiles.