HistoTx: Early fusion of H&E images and spatial transcriptomics at varying spatial transcriptomics resolution with self-supervised learning

AI summary

Overview

Research area: Computational pathology and multimodal machine learning — specifically, joint modeling of hematoxylin and eosin (H&E) whole slide images with spatial transcriptomics (ST) data.

Technical level: Advanced. The paper assumes familiarity with vision transformers, self-supervised learning (DINOv2), contrastive alignment methods such as CLIP/SEAL, and spatial transcriptomics platforms (Visium, Xenium).

Scope in one sentence: HistoTx is an early-fusion vision transformer that merges transcript tokens — built at flexible spatial resolutions — with image patch tokens inside a single shared transformer stack, trained by continuing DINOv2 self-supervised pretraining on paired image–transcript data from a vision-only pathology foundation model.

What This Paper Is About

Most existing methods that combine H&E images with spatial transcriptomics use the molecular data only as a training signal — through contrastive alignment or supervised fine-tuning — so at inference the model is fundamentally image-driven, and transcriptomics is never a true input. HistoTx instead treats transcripts as an actual input modality: transcript tokens are constructed at arbitrary spatial resolution, concatenated with image patch tokens, and processed jointly so the two modalities interact deeply inside the transformer. The goal is a single model that keeps strong image-only performance, improves on tasks that need molecular knowledge (like gene expression prediction), and gains further when both modalities are available at inference.

Key Contributions

1. An early-fusion multimodal architecture. HistoTx builds transcript tokens at arbitrary spatial resolution (decoupled from the image token grid), projects them into the image token dimension, applies the morphology encoder's positional embeddings (with dynamic spatial resampling when token counts differ), and passes the fused sequence [CLS | image tokens | transcript tokens] through a shared ViT stack for deep cross-modal interaction.

2. A multimodal self-supervised pretraining strategy. The authors continue DINOv2 pretraining on paired image–ST data, warm-started from a vision-only pathology foundation model. Transcripts are included only in the DINO global views and omitted from the local views, so the standard DINO local-crop loss on CLS tokens becomes a cross-modality loss that distills molecular knowledge into the visual encoder.

3. A flexible transcript tokenization scheme spanning two ST platforms. Xenium transcripts are counted per gene within non-overlapping T×T pixel regions (yielding 256, 64, 16, or 1 tokens for T = 14, 28, 56, or 256 on a 224×224 image), while Visium spots contribute one aggregated token each. This lets a single backbone train on both probe-based and spot-based data, enabling training on the full HEST-1k dataset.

4. A broad downstream evaluation. The model is benchmarked on gene expression prediction (HEST-Bench, HEST-Expr, ETOP BEAT-meso), image-only classification (Camelyon16, BACH, BRACS, BreakHis, Gleason TMA, PANDA Radboud), and segmentation (MoNuSAC, Tan Tock Seng, CoNSeP), against an image-only baseline and SEAL with a CONCH backbone.

Main Findings

• Gene expression prediction improves most. On HEST-Bench (predicting 50 highly variable genes from H&E patches), mean Pearson correlation rises from 0.31 for the image-only baseline to 0.38 for HistoTx, versus 0.35 for SEAL+CONCH. The authors note that part of this gain may come from increased exposure to HEST images during continued pretraining, not only multimodal pretraining.

• A supervised task-specific baseline still wins on BEAT. On the ETOP BEAT-meso task (predicting 100 Xenium add-on genes), HistoTx reaches 0.66 Pearson correlation versus 0.53 for the image-only BEATIO baseline, but a supervised baseline trained directly for that task reaches 0.82.

• Multimodal inputs at inference beat either modality alone. HEST-Expr reaches 0.36 with 1,000 input genes (image-only HEST-Expr variant: 0.26 baseline, 0.30 with HistoTx), and BEAT reaches 0.66 with 380 input genes (image-only BEATIO: 0.53 baseline, 0.58 with HistoTx).

• Gains are clearest on regression and segmentation. The paper reports improvements of 0.01 to 0.08 points over the image-only base on HBench, HexprIO, BEATIO, and on CoNSeP (0.61→0.62), MoNuSAC (0.61→0.62), and Tan Tock Seng (0.31→0.37). Camelyon16 binary balanced accuracy rises from 0.79 to 0.87.

• Breast cancer subtype classification degrades. BACH drops from 0.88 to 0.82, BRACS from 0.64 to 0.60, and BreakHis from 0.82 to 0.73, while prostate results are mixed (PANDA Radboud 0.61→0.67, Gleason TMA 0.77→0.69). The authors hypothesize that continued multimodal pretraining reorients the encoder toward gene-expression-aligned features and weakens subtle morphological detail needed for visually similar subtypes, and state that whether this is tissue-specific or task-specific remains open.

• More expression tokens help slightly. In the appendix tables, the 256-token variant edges out the 16-token variant on several tasks (e.g., HEST-Bench 0.38 for both, Camelyon16 0.92 vs. 0.87, MoNuSAC 0.64 vs. 0.62, average 0.67 vs. 0.66), while the image-only baseline average is 0.65.

• The chosen default is a deliberate middle ground. The main text focuses on the variant with one expression token per Visium spot and 16 expression tokens per 224×224 Xenium image, on the reasoning that very fine transcript tokens are noisy while very few sacrifice spatial detail.

Methodology in Plain English

The starting point is an existing vision-only pathology model (a ViT-B/14 backbone initialized from an in-house DINOv2 "Midnight" checkpoint) that already understands H&E histology. The authors add a second input stream: gene expression. Each expression measurement is turned into a token either by counting gene detections in small square regions of the image (for Xenium) or by taking the aggregated expression profile of a Visium spot. Those count vectors are encoded by a frozen Geneformer model (gf-6L-10M-i2048), batch-normalized, linearly projected to the image token dimension, and rescaled to match the magnitude of image tokens. The image patches and transcript tokens are then glued into one sequence with a CLS token in front and pushed through the same transformer blocks, so image and molecular information interact at every layer rather than being concatenated at the end.

Training is self-supervised, using the DINOv2 recipe (plus the iBOT masked-image-modeling auxiliary loss). The trick that produces cross-modal learning: DINO creates two large "global" views and six smaller "local" views of each sample. Transcripts are attached only to the global views, so the global CLS token is multimodal while the local CLS tokens are purely visual. The standard DINO loss that forces local and global CLS tokens to agree therefore pushes the image encoder to absorb molecular information.

Training mixes six data streams at roughly equal weight (17% each): BEAT-meso images, BEAT-meso images with Xenium transcripts, HEST Xenium with spatial transcripts, TCGA image-only, HEST image-only, and HEST spot-level expression. Data includes 7,924 TCGA diagnostic slides; HEST-1k v1.1.0 (1,229 WSIs, 26 organs, 65 Xenium samples, 1,164 spot-based samples); and ETOP BEAT-meso (163 mesothelioma patients, 122 train / 41 test). Optimization uses AdamW with a base learning rate of 2×10⁻⁴, cosine schedules, BF16 mixed precision, an effective batch size of 1,024, and at most 200 epochs. Evaluation uses linear probing for classification, lightweight decoder probes for segmentation, and Ridge regression on frozen CLS embeddings for the BEAT gene prediction tasks.

Why This Matters

Impact on research: The paper challenges the dominant assumption that spatial transcriptomics should serve only as a training signal for image encoders. By demonstrating that transcripts can be first-class inputs inside a shared transformer — with benefits visible both when they are present and, on several tasks, when they are absent — it reframes the design space for multimodal pathology models. The flexible tokenization also removes a practical barrier: most prior models handle either spot-based or probe-based ST, and the authors state that only MINT can encode both with one model, limited to one learnable ST token.

Real-world applications:

• Improving gene expression inference from routine H&E slides, where molecular assays are unavailable — relevant to biomarker discovery and tumor characterization in low-resource settings.
• Panel imputation: the HEST-Expr and BEAT tasks frame a concrete clinical problem, predicting genes outside a measured Xenium panel (e.g., a 380-gene immuno-oncology panel plus 100 custom genes) from the genes that were measured.
• Tumor subtyping and grading support, where morphology alone can be ambiguous but molecular context helps — though the paper's own breast cancer results show this benefit is not automatic.
• Multimodal quality control and research tooling for studies that already collect both H&E and ST, letting one backbone serve both image-only archives and paired cohorts.

Industry relevance: The approach lets a single backbone be pretrained once and deployed either image-only (cheap, ubiquitous H&E) or image-plus-transcript (when an ST assay is available), which matches how pharmaceutical and diagnostics pipelines are actually staged. The retention of image-only performance on tasks like CRC-100K (0.96) and PatchCamelyon (0.89–0.90) matters because a multimodal model that degrades on standard pathology benchmarks would be hard to justify. The authors also explicitly motivate the image-centric prior by the cost of ST versus the near-universal availability of H&E.

Future Directions

• Diagnose and mitigate the subtype-classification regression. The drops on BACH, BRACS, and BreakHis are unexplained beyond a hypothesis; disentangling tissue-specific from task-specific effects — and testing whether reweighting or adapter-style tuning recovers subtype accuracy — is the obvious next step.
• Narrow the gap to supervised, task-specific baselines. On BEAT, a supervised model trained directly for the task reaches 0.82 versus 0.66 for HistoTx, suggesting room for task-adaptive fine-tuning or better fusion of input expression panels.
• Extend and ablate the tokenization and resolution grid. Only two variants (16 and 256 tokens) appear in the reported configuration tables; the effect of jointly varying transcript resolution and image resolution (mpp values of 0.25, 0.5, 1.0, 2.0) on downstream utility is not fully characterized in the provided content.
• Scale the backbone and data. All results use a ViT-B/14 backbone and a family of variants; whether the early-fusion advantage grows with larger encoders, more Xenium samples, or additional ST platforms is untested here.

Target Audience

Machine learning researchers working on multimodal biomedical foundation models; computational pathologists and bioinformaticians interested in integrating spatial transcriptomics with histology; and industry or translational teams building pathology models that must operate both with and without molecular data. Readers need a working understanding of vision transformers, self-supervised learning objectives, and ST platform differences (Visium spots versus Xenium transcript detections) to follow the architecture and evaluation details.

Authors’ abstract

Spatial transcriptomics maps gene expression across tissue sections and, when integrated with whole slide images, jointly captures both morphological architecture and molecular activity to provide a rich, multimodal view of tissue biology. ML approaches combining spatial transcriptomics and WSIs typically learn a mapping from morphological features to molecular profiles via contrastive alignment or supervised fine-tuning of pathology foundation models. In both cases, transcriptomics serves as a training signal rather than a true input modality, leaving inference fundamentally image-driven. We propose an early-fusion vision transformer architecture in which transcript tokens are constructed at arbitrary spatial resolution, merged with image patch tokens, and jointly processed through a shared transformer stack for deep cross-modal interaction. Starting from a pretrained vision-only pathology foundation model, we train HistoTx by continuing self-supervised pretraining on paired image and spatial transcriptomics data and demonstrate that: 1) at inference with image only, performance remains on par with the baseline on most tasks while improving on tasks that inherently benefit from molecular knowledge, such as gene expression prediction; 2) when both modalities are available at inference, jointly providing image and transcript inputs outperforms either modality alone.