ConSurv: Multimodal Continual Learning for Survival Analysis

AI summary

Overview

Research area: Multimodal machine learning for computational pathology and cancer survival prediction, specifically continual learning with whole slide images (WSIs) and genomics.

Technical level: Advanced. The paper builds on mixture-of-experts architectures, knowledge distillation, and replay-buffer continual learning, and assumes familiarity with the C-index and survival analysis.

Scope: The paper proposes ConSurv, a multimodal continual learning method for survival prediction across four cancer datasets, along with a new incremental-learning benchmark called MSAIL.

What This Paper Is About

Survival prediction models are usually trained on one dataset and then frozen, so they cannot adapt as new cancer data arrives without expensive retraining. Continual learning can update a model on new datasets sequentially, but existing continual learning methods were designed for a single data modality and suffer severe catastrophic forgetting on survival prediction; they also ignore the relationship between WSIs and genomic data. The paper's goal is to build a method that learns from a stream of multimodal cancer datasets while retaining what it learned earlier.

Key Contributions

1. ConSurv, described as the first multimodal continual learning (MMCL) method for survival analysis across multiple cancers using WSI and genomic data.
2. MS-MoE (Multi-staged Mixture of Experts), an expandable module that captures task-shared and task-specific knowledge at three stages: the WSI encoder, the genomic encoder, and the modality fusion component.
3. FCR (Feature Constrained Replay), which limits feature deviation at encoder level for both modalities and at the fusion level, using a fixed-size replay buffer of stored feature representations rather than raw WSIs and genomics.
4. The MSAIL benchmark (Multimodal Survival Analysis Incremental Learning), built from four TCGA datasets: BLCA, UCEC, LUAD, and BRCA.

Main Findings

• Static models transfer only partially. A model trained on BLCA reaches a C-index of 0.607, and before any training on UCEC it scores 0.545 on UCEC, above the random baseline of 0.420. After training on UCEC the score rises to 0.648, showing that positive forward transfer exists but is well below what training achieves.
• Catastrophic forgetting is severe under sequential finetuning. BLCA performance drops from 0.607 to 0.531 after the model is subsequently trained on UCEC, LUAD, and BRCA.
• ConSurv leads on the main metrics. On average C-index, ConSurv reaches 0.601±0.045 versus 0.572±0.024 for finetuning and 0.611±0.037 for the joint-training upper bound. On average C-index IPCW it reaches 0.597±0.039, versus 0.528±0.055 for finetuning and 0.545±0.045 for joint training.
• Some unimodal CL methods underperform finetuning. The paper reports that several compared CL methods score below finetuning, which it attributes to neglecting inter-modal interactions.
• ConSurv attains the best BWT and FWT for C-index IPCW among compared methods (BWT 0.002±0.080, FWT 0.083±0.103). The paper notes its other metrics are not the highest, which it explains by the trade-off between absolute performance and resistance to forgetting.
• Both modules help individually. MS-MoE alone raises average C-index from 0.572±0.024 to 0.585±0.022 and C-index IPCW from 0.528±0.055 to 0.575±0.015. FCR alone raises them to 0.581±0.043 and 0.545±0.045. Combining both gives 0.601±0.045 and 0.597±0.039.
• Fusion-level constraint alone is already strong. Using FCR with only the final fusion representation (marked "✓(f)" with MS-MoE) yields 0.599±0.026 average C-index, and adding the patch and genomics feature constraints improves further to 0.601±0.045.
• Expert routing differentiates tasks. In the patch encoder's MS-MoE, expert E3 specializes in BRCA; in the genomic encoder's MS-MoE, expert E6 is selected for UCEC and BRCA. Expert E8 acts as the shared expert and is always selected. The random-selection reference line is 2/7.
• Risk stratification is statistically significant. Kaplan–Meier analysis on all four datasets separates low-risk and high-risk groups with log-rank p-values below 0.05, the paper's stated significance convention.

Methodology in Plain English

The researchers take an existing state-of-the-art multimodal survival model, MoME, as the backbone and train it on a sequence of cancer datasets: BLCA, then UCEC, then LUAD, then BRCA. Two additions protect against forgetting.

First, they insert mixture-of-experts blocks at three points in the network, after the WSI encoder, after the genomic encoder, and inside the fusion component. Each block keeps a fixed pool of experts and, when a new dataset arrives, adds only a new lightweight router that decides which experts to use. Selection is sparse (a top-k choice) plus one shared expert that is always active, which is meant to encourage both task-specific and shared knowledge while limiting parameter growth. Blocks are either swapped in for an existing feed-forward or linear layer ("replace" mode) or added with a residual connection ("append" mode), so the original backbone behavior is approximately recoverable.

Second, since storing gigapixel WSIs and genomic data is expensive, they store only a small fixed number of processed feature representations, chosen by reservoir sampling so every seen instance has an equal chance of being retained. During training on a new dataset, the model is penalized for drifting away from those stored features at three levels: patch features, genomic features, and the final fused representation. It also replays the stored samples with their ground-truth labels. The overall objective combines the current dataset's survival loss with these feature-constraint and replay terms, weighted by hyperparameters alpha and beta.

Evaluation uses C-index and C-index IPCW, averaged across datasets, plus Forgetting, Backward Transfer, and Forward Transfer for reference. The benchmark is run in a task-incremental setting where task identities are available at inference.

Why This Matters

Research impact: The paper introduces both a method and a benchmark for a setting the authors state had no prior CL studies: WSI plus genomics continual learning. It also shows that unimodal CL baselines do not transfer straightforwardly to this multimodal regime, motivating dedicated MMCL research. The code is released at https://github.com/LucyDYu/ConSurv.

Real-world applications:

• Updating cancer prognosis models as new patient cohorts are collected, without retraining from scratch on all prior data.
• Maintaining performance across different cancer types within one deployed model, rather than one static model per cancer.
• Adapting to shifts caused by changes in staining protocols, imaging technology, and genomic sequencing quality, which the paper cites as reasons static models become outdated.
• Supporting treatment planning and mortality-risk communication through risk stratification of patients into low-risk and high-risk groups.

Industry relevance: The approach targets a practical pain point for clinical and health-AI organizations, namely the compute and resource cost of repeatedly retraining on combined old and new data. Because FCR stores features instead of raw WSIs and genomics, storage overhead is reduced relative to naive data replay, which matters given the size of gigapixel slides.

Future Directions

• Interpret the detailed cost and limitation analysis. The paper states that an analysis of computational costs, limitations, and future work appears in Appendix D, but that appendix is not included in the content provided here.
• Extend beyond four TCGA cancer types. The benchmark covers BLCA, UCEC, LUAD, and BRCA; whether the approach scales to longer dataset sequences and more modalities is untested in the reported results.
• Investigate the stability-plasticity trade-off further. ConSurv wins on average C-index and average C-index IPCW but not on every reference metric, so the paper's own reported trade-off invites closer study of how alpha and beta should be set.
• Test generalization beyond the task-incremental setting. The paper focuses on task-incremental learning with task identities available at inference; class-incremental or domain-incremental variants for WSI and genomics are not evaluated.

Target Audience

Researchers and practitioners in medical image analysis, computational pathology, and multimodal machine learning who are interested in continual learning for clinical prediction. It is also relevant to clinical AI engineers who deploy and maintain prognostic models that must be updated as new data arrives, and to benchmark builders who need a reference protocol for multimodal incremental survival analysis. Graduate students with background in survival analysis and deep learning will find it accessible with some effort, but the mixture-of-experts and distillation components make it suited to an advanced audience.

Authors’ abstract

Survival prediction of cancers is crucial for clinical practice, as it informs mortality risks and influences treatment plans. However, a static model trained on a single dataset fails to adapt to the dynamically evolving clinical environment and continuous data streams, limiting its practical utility. While continual learning (CL) offers a solution to learn dynamically from new datasets, existing CL methods primarily focus on unimodal inputs and suffer from severe catastrophic forgetting in survival prediction. In real-world scenarios, multimodal inputs often provide comprehensive and complementary information, such as whole slide images and genomics; and neglecting inter-modal correlations negatively impacts the performance. To address the two challenges of catastrophic forgetting and complex inter-modal interactions between gigapixel whole slide images and genomics, we propose ConSurv, the first multimodal continual learning (MMCL) method for survival analysis. ConSurv incorporates two key components: Multi-staged Mixture of Experts (MS-MoE) and Feature Constrained Replay (FCR). MS-MoE captures both task-shared and task-specific knowledge at different learning stages of the network, including two modality encoders and the modality fusion component, learning inter-modal relationships. FCR further enhances learned knowledge and mitigates forgetting by restricting feature deviation of previous data at different levels, including encoder-level features of two modalities and the fusion-level representations. Additionally, we introduce a new benchmark integrating four datasets, Multimodal Survival Analysis Incremental Learning (MSAIL), for comprehensive evaluation in the CL setting. Extensive experiments demonstrate that ConSurv outperforms competing methods across multiple metrics.