Gene Incremental Learning for Single-Cell Transcriptomics

AI summary

Overview

• Research area: Machine learning for single-cell transcriptomics (single-cell RNA sequencing), specifically incremental/continual learning applied to genes treated as tokens.
• Technical level: Intermediate. The paper assumes familiarity with incremental learning concepts (catastrophic forgetting, replay, distillation) and with Transformer-based models of gene expression, but it introduces its framework from first principles.
• Scope: The paper defines a Gene Incremental Learning (GIL) framework for single-cell transcriptomic data, proposes two evaluation protocols, adapts existing class-incremental-learning methods to it, and releases a benchmark for the task. (arXiv:2511.13762v1 [cs.LG], 14 Nov 2025; accepted by AAAI 2026.)

What This Paper Is About

Incremental learning has been studied extensively for classes in Computer Vision, but hardly at all for tokens, even though token sets also grow over time (new words in language, newly measured genes in biology). Tokens are hard to split into incremental stages in language because text is holistic — removing a word changes or destroys the meaning of a sentence. The authors point out that genes in single-cell transcriptomics do not have this problem, because transcriptomic data has no relative order among genes, so genes can be freely partitioned and rearranged across stages. Their goal is to build that incremental pipeline, define how to evaluate it, and check whether existing class-incremental-learning methods can reduce gene forgetting.

Key Contributions

1. A Gene Incremental Learning (GIL) framework defined over single-cell transcriptomic datasets, which addresses the previously unstudied problem of incremental learning under continuous growth of genes. The design introduces base genes that persist in every stage plus stage-specific gene subsets that are mutually exclusive across stages, because genes — unlike classes — do not map one-to-one onto samples (every sample contains all genes) and a single gene cannot convey the meaning of a whole sample.
2. Two evaluation protocols for GIL: a gene-wise regression metric that measures forgetting directly through masked gene value prediction loss, and a gene-based classification metric that selects genes crucial to particular downstream classification datasets and places them in different stages, so downstream accuracy indicates whether those genes were memorized or forgotten.
3. Adaptation of class incremental learning methods to GIL — data replay (gene replay) and knowledge distillation (gene distillation) — together with baseline and oracle references, validated through extensive experiments.
4. A complete benchmark for Gene Incremental Learning in single-cell transcriptomics, released as code at https://github.com/simpleshinobu/scbenchmark.

Main Findings

• Gene forgetting is real. In the absence of any knowledge-preserving method, the baseline progressively forgets previously learned genes: in the 2-stage GIL settings performance drops by 0.279 in regression and 1.816% in downstream classification on average, and drops by 0.181 in regression in the 3-stage GIL setting.
• The two evaluations agree. Both gene-wise regression and gene-based classification show declines as stages increase when evaluating the baseline, which the authors present as evidence that the proposed evaluations are consistent.
• Adapted CIL methods work. Gene replay and gene distillation both improve gene-wise regression across settings. Best observed averaged Δ values were 0.018 for gene replay and 0.154 for gene distillation within the swept hyperparameter range.
• Replay is the stronger method overall. In the 2-stage regression results, replay achieved averaged Δ of 0.037 versus 0.212 for distillation and 0.279 for the baseline. In the 3-stage setting, replay reached a stage-3 Δ of 0.028 versus 0.177 for distillation and 0.181 for the baseline.
• Replay scales as expected. In the Norman-Lupus ablation, increasing replayed samples from 50 to 10² to 10³ to 10⁴ improved Δ from 0.121 to 0.091 to 0.043 to 0.018; the paper notes that as replay grows it degenerates toward the oracle method.
• Distillation has a trade-off. Increasing λ from 0.5 to 1.0 to 5.0 to 10.0 improved Δ from 0.248 to 0.230 to 0.193 to 0.154, but the loss on the newly learned Lupus genes worsened, rising from 0.134 to 0.143, showing that learning of stage 2 gradually fails as more previous knowledge is retained.
• Distillation is inconsistent across the two evaluations. It achieved an average Δ of 0.212 in gene-wise regression, better than the baseline's 0.279, but its average drop in gene-based classification was 2.473%, worse than the baseline's 1.816%. The authors suggest distillation prevents average forgetting but degrades the features of learned genes, and argue that only methods improving on all evaluations are robust.
• Oracle accuracies for the same dataset differ between settings because crucial genes shared across downstream datasets were removed to prevent cross-contamination. For example, although both the Norman-Lupus and Lupus-Panc settings include Lupus, the selected crucial genes for Lupus differ.
• Few settings are possible. Transcriptomic downstream datasets share large amounts of overlapping crucial genes; the baseline reaches 67.313% on Lupus at the first stage partly because some base genes and specific genes for Norman help its classification. If a gene crucial to many datasets were assigned to one specific stage, gene-based classification would fail.
• Measuring forgetting is harder in GIL than in CIL. A class as a label strongly drives classification and new classes are exclusive from old ones, so forgetting is easily observed. A gene is only a small part of the input, so even an unlearned gene may not affect downstream tasks, and the performance drops in the classification results remain modest.

Methodology in Plain English

The authors start from how class incremental learning works: a model trains on a set of classes, then later stages add new classes while old data is unavailable, and the model is tested on all classes seen so far. They transplant this idea onto genes by exploiting a key property of transcriptomic data — genes, unlike words in a sentence, have no inherent order, so they can be split up freely.

Their setup keeps a fixed set of base genes present in every stage so that samples remain biologically meaningful. The remaining genes are split into mutually exclusive subsets, one per stage. Datasets are partitioned separately, because each sample contains all genes, so splitting genes does not automatically split the data. In each stage the model only sees the base genes plus that stage's specific genes, and it is evaluated on everything seen so far.

For learning, they follow the standard masked value prediction objective used in Transformer-based transcriptomics models: mask some gene expression values and predict them. They show the model binds genes to their values through an embedding layer, a value-encoding linear layer, a Transformer backbone, and a prediction linear layer, so learning values amounts to learning genes.

They define a baseline that trains only on the current stage, and an oracle trained on all splits jointly as an upper reference. They then adapt two families of class incremental learning methods: gene replay, which retains a subset of samples from earlier stages, and gene distillation, which adds a penalty for deviating from the previous stage's optimal model outputs, with the current stage's specific genes removed from the distillation term because the old model could not predict unseen genes.

For evaluation, they average masked value regression loss over the specific genes of each stage, and separately they pick genes that are crucial to particular downstream classification datasets, spread those genes across stages, freeze the trained feature extractor, and train only a single linear layer to measure classification accuracy.

Why This Matters

• Research impact: It opens incremental learning for tokens, not just classes, and shows that single-cell transcriptomics is a natural testbed for it. It provides the first benchmark, baselines, oracle references, and metrics for GIL, giving later work a common ground.

• It reframes a practical biological problem as a machine learning problem. Gene panels and measurement technologies expand over time, so models that must absorb new genes without discarding old ones are a real operational need.

• Real-world applications:

  • Analyzing newly discovered or newly assayed genes alongside previously measured ones without retraining models from scratch.
  • Refining disease-relevant gene panels (the paper's downstream tasks span Lupus, pancreas, colitis, myeloid pan-cancer analysis, and perturbation studies) as panels evolve.
  • Maintaining large pretrained single-cell models whose gene vocabularies grow with new studies and technologies.
  • Diagnosing cell types across many organs and tissues (the underlying data collection covers over 50 organs and tissues, such as blood and heart, from more than 400 studies).

• Industry relevance: The work touches foundation-model maintenance for life sciences and pharma, where models pretrained on large gene vocabularies need to incorporate new genes. The benchmark, code release, and the finding that simple replay works well lower the barrier to adopting incremental strategies in production single-cell pipelines. The paper also notes broader relevance to any field where tokens accumulate, including natural language processing vocabularies.

Future Directions

• Design GIL-specific algorithms. The authors state as future work that they will try to design methods tailored to GIL, rather than only adapting existing class incremental learning methods.
• Extend the framework to other token-learning fields. They aim to carry the framework beyond transcriptomics, including to domains such as language where token growth also occurs.
• Develop better evaluations for gene forgetting. The paper argues that detecting forgetting of a gene is intrinsically difficult because a single gene is only a small part of the input, so stronger evaluation protocols are needed.
• Broaden the range of benchmark settings. Only a small number of settings and a limited number of stages were possible, because downstream datasets share too many crucial genes; the authors note that selecting valid settings required preliminary experiments, which leaves room for more settings.

Target Audience

Researchers and practitioners working on continual or incremental learning who want to move beyond classes to tokens; computational biologists and bioinformaticians working with single-cell RNA sequencing and Transformer-based models such as the scGPT-style pipeline this work builds on; and machine learning engineers maintaining growing gene vocabularies for foundation models in the life sciences. Readers need a basic grasp of incremental learning and of applying Transformer models to gene expression data; the paper's own framework is explained from the ground up.

Authors’ abstract

Classes, as fundamental elements of Computer Vision, have been extensively studied within incremental learning frameworks. In contrast, tokens, which play essential roles in many research fields, exhibit similar characteristics of growth, yet investigations into their incremental learning remain significantly scarce. This research gap primarily stems from the holistic nature of tokens in language, which imposes significant challenges on the design of incremental learning frameworks for them. To overcome this obstacle, in this work, we turn to a type of token, gene, for a large-scale biological dataset--single-cell transcriptomics--to formulate a pipeline for gene incremental learning and establish corresponding evaluations. We found that the forgetting problem also exists in gene incremental learning, thus we adapted existing class incremental learning methods to mitigate the forgetting of genes. Through extensive experiments, we demonstrated the soundness of our framework design and evaluations, as well as the effectiveness of our method adaptations. Finally, we provide a complete benchmark for gene incremental learning in single-cell transcriptomics.