spora: A Unified Multimodal Dataset and Platform for Spatial Proteomics

AI summary

Overview

Research area: Computational pathology and spatial proteomics, at the intersection of medical imaging, multimodal machine learning, and biomedical foundation models.

Technical level: Intermediate. The benchmark results and dataset descriptions are accessible to a general ML reader; the tiling algorithm and standardization routines assume some familiarity with optimization and image preprocessing.

Scope: This paper introduces spora, a harmonized dataset of 13,546 multiplex spatial proteomics images from 5,477 patients across 35 cohorts, together with a data-loading library (spora[io]) and a benchmark suite (spora[bench]) for evaluating foundation models on cell- and tissue-level tasks.

What This Paper Is About

Spatial proteomics measures dozens of proteins across intact tissue sections at single-cell resolution, but every study uses different antibody panels, imaging platforms, file formats, and annotation conventions. This fragmentation means no single study provides enough diverse data to train a general-purpose model, and no public resource has consolidated the field the way TCGA did for histopathology or CELLxGENE did for single-cell RNA sequencing. The authors build a unified, AI-ready corpus plus the tooling and benchmarks needed to train and compare foundation models across cohorts, technologies, and tissue types.

Key Contributions

1. spora[data] — a unified spatial proteomics dataset. 13,546 standardized multiplex images from 5,477 unique patients across 35 cohorts, 13 tissue origins, 295 markers, and five technologies (IMC, CODEX, Orion, MIBI, IF). All images are resampled to 1.0 µm/px and stored as .ome.zarr, with tissue masks, cell instance masks, structured patient/clinical/acquisition metadata, paired H&E where available, and per-cell phenotype annotations for 16 cohorts.

2. spora[io] — a multimodal loading library. A Python package providing tissue-aware tile coverage (posed as a budgeted maximum coverage problem with a (1−1/e) ≈ 0.63 greedy approximation guarantee), cross-cohort weighted sampling, lazy multimodal loading of multiplex images, H&E, and masks, and standardization at channel, tissue, cohort, and global levels.

3. spora[bench] — a benchmark suite for oncology. Eight segmentation, 18 cell phenotyping, six diagnostic/therapeutic prediction, and eight virtual staining tasks across twelve cohorts, with fixed patient-level 80/20 train–test splits and a modular wrapper for plugging in new foundation models.

4. The first head-to-head comparison of spatial proteomics foundation models (VirTues and KRONOS) against task-specific baselines on a common protocol, including disclosure practices for pretraining exposure.

Main Findings

• Cell phenotyping favors VirTues. On linear-probe cell classification across ten cohorts and 12.4 million cells, VirTues achieved the highest macro F1 in every cohort, ranging from 0.611 to 0.836, while KRONOS ranged from 0.355 to 0.601 (no KRONOS result is reported for Cords et al. 2024). On cross-cohort annotation across eight cohorts harmonized into seven classes, VirTues ranked first on four cohorts and second on three, ahead of astir and MAPS.

• Segmentation generalizes unevenly. In zero-shot cell instance segmentation at IoU threshold τ = 0.5 over eight cohorts, VirTues led on six of eight cohorts, often by a substantial margin; Cellpose accounted for the remaining two.

• Tissue-level clinical signal is present but unsaturated. Across six diagnostic and therapeutic tasks spanning non-small cell lung cancer, triple-negative breast cancer, and a heterogeneous breast cancer cohort (3,284 tissues from 2,002 patients), VirTues outperformed KRONOS on four tasks and matched it on patient risk stratification and breast cancer grading. Foundation models beat classical baselines across cell-, tissue-, and patient-level tasks, but the authors state substantial headroom remains.

• Multiplex-to-multiplex virtual staining works best with VirTues. Across four cohorts, VirTues predictions correlated consistently well with ground truth and outperformed Eva, a per-marker linear model, and a nearest-marker baseline. Performance varied by marker and tracked the nearest-marker baseline, suggesting some markers are redundant.

• H&E-to-multiplex virtual staining is far from solved. Three dedicated models (GigaTIME, HistoPlexer, ROSIE), trained on closed-source corpora of 21 whole-slide images, 336 regions of interest, and 1,342 tumor cores respectively, produced correlations near zero for most markers across four held-out cohorts.

• Tiling matters. The tissue-aware tiler reached tissue-to-background ratios of 5.43 to 6.18 on the tissues shown, well above standard grid tiling, by adapting to irregular tissue geometry and interior voids.

• Coverage is biased. The authors acknowledge spora is skewed toward cancer and covers only a subset of cancer types and organs, reflecting where the field has concentrated effort.

Methodology in Plain English

The authors spent over two years collecting publicly released spatial proteomics cohorts under permissive licenses, either from open repositories or directly from original study authors. Rather than simply aggregating files, they harmonized them: folder structures, file formats, and metadata schemas were unified, marker names were mapped to consistent names and UniProt identifiers, nuclear stains were flagged with a dedicated column so segmentation tools can find them, and all images were resampled to a common resolution and stored in a standard chunked format that supports lazy loading. Cell segmentations were recomputed with a single model (VirTues), chosen after comparison with alternatives and expert review, while original authors' masks were retained for reference. Each tissue received a globally unique identifier that also encodes whether its H&E image was pixel-registered to the same slide.

For the loading library, the team reframed tiling as an optimization problem: choose a fixed number of tiles that cover as much foreground tissue as possible, subject to a purity constraint. Because this is NP-hard but the objective is submodular, a greedy procedure gives a guaranteed approximation, and the implementation uses summed-area tables and local updates to stay practical on whole-slide images. Tile coordinates are precomputed and released so results are reproducible across papers. For benchmarking, each foundation model is wrapped in a common interface, cell representations are extracted per the model's own prescribed procedure, and downstream classifiers (logistic regression probes, or attention-based multiple instance learning for tissue-level tasks) are trained on fixed patient-level splits.

Why This Matters

Research impact. Spatial proteomics has lagged behind histopathology and spatial transcriptomics in data consolidation, and the paper argues this is precisely what has blocked foundation model development for the modality. By providing harmonized data, a stable IO interface, and shared splits, spora lets researchers compare models without confounding differences in preprocessing, and turns dozens of bespoke single-study problems into a shared benchmark. The dataset is hosted by EMBL-EBI's BioImage Archive.

Real-world applications:

• Immunotherapy response prediction and risk stratification, using tumor microenvironment features such as immune cell phenotypes and checkpoint expression.
• Virtual staining, imputing unmeasured protein channels from H&E or from a reduced marker panel, which could extend an assay's effective panel size without consuming additional tissue.
• Cross-cohort cell annotation, automatically labeling newly acquired cohorts for which no expert annotations exist.
• Cancer subtyping and grading from multiplex tissue images, supporting diagnostic decision-making.

Industry relevance. Pharma and diagnostics companies working on spatial biology, companion diagnostics, and tissue-based biomarkers depend on cross-platform comparability; a standardized corpus and benchmark lowers the cost of validating models across scanners, panels, and cohorts, and gives a common yardstick for vendor and model claims.

Future Directions

• Broadening beyond cancer. The authors explicitly note the corpus is cancer-skewed and covers only a subset of cancer types and organs; expanding to inflammatory, metabolic, and other diseases is a natural next step, along with community-contributed cohorts.

• Closing the H&E-to-multiplex gap. Current virtual staining models trained on small closed-source corpora show near-zero correlations on the new benchmark cohorts; cross-cohort, cross-platform training is the stated route to improving them.

• Architecture and aggregation research. The authors attribute part of the remaining performance gap to architectural and aggregation choices that a dataset alone cannot fix, pointing to model design as an open problem.

• Building spatial tissue atlases. The paper frames spora as groundwork for reference maps of how cells organize within tissues across health and disease, which would provide reference frames for interpreting new patient samples.

Target Audience

Machine learning researchers working on medical imaging and multimodal foundation models; computational pathologists and spatial biology method developers; bioinformaticians who need a standardized, programmatically accessible spatial proteomics corpus; and translational oncology groups evaluating tissue-based biomarkers. Readers focused on optimization and data pipeline design will also find the tiling formulation and standardization strategy relevant.

Authors’ abstract

Spatial proteomics (SP) captures the molecular composition and spatial organization of tissues at single-cell resolution, opening a direct view of the cellular ecosystems that drive disease. Protein panels, acquisition platforms, and tissue contexts differ from one cohort to the next, and a coherent picture of tissue biology can only emerge from models trained to bridge this heterogeneity. Foundation models offer the natural path forward, but their training depends on large, harmonized corpora that span platforms, panels, and cohorts. No public resource currently approaches that scale; available data sit in small, narrowly scoped releases with incompatible formats and conventions. We present $\texttt{spora}$, a large-scale multimodal SP dataset of 13,546 standardized multiplex images from 5,477 patients across 35 cohorts and five major SP technologies. Curated with biomedical experts, $\texttt{spora}$ provides model training-ready formats together with cell and tissue segmentations, structured clinical metadata, and, where available, paired H&E enabling the training of virtual staining models. It is complemented by $\texttt{spora [io]}$, a unified interface for tiling, sampling, and standardization across all cohorts and modalities, and $\texttt{spora [bench]}$, a benchmark suite spanning cell phenotyping, diagnostic and therapeutic prediction as well as virtual staining tasks across twelve cohorts that places current SP foundation models on common ground against task specific baselines.