MUSS: A Multi-scale and Sequence-based Model for Single-cell Gene Regulation

AI summary

Overview

Research area: Computational genomics and single-cell machine learning, specifically sequence-based modeling of gene regulation that links DNA sequence, chromatin accessibility (scATAC-seq), and gene expression (scRNA-seq).

Technical level: Advanced. The paper assumes familiarity with transformer architectures (self-attention, cross-attention, rotary positional encodings, masked language modeling), single-cell multi-omics assays, and regulatory genomics benchmarks.

Scope (one sentence): The paper introduces MUSS, a model that derives peak- and gene-level representations directly from DNA sequence using a frozen AlphaGenome encoder and integrates them with cell-specific accessibility through multi-scale attention, then evaluates it on gene expression prediction, enhancer–gene linking, motif-guided prioritization, and zero-shot cell clustering.

What This Paper Is About

Cells in the same body share one genome but express different genes, because transcription factors bind accessible regulatory regions and act on target genes across distances spanning kilobases to megabases. Two families of models exist: sequence-based models that read DNA but lack single-cell resolution, and single-cell omics models that represent genes and peaks as fixed-vocabulary tokens without their underlying sequence. MUSS aims to unify both, producing representations for peaks and genes that are grounded in DNA sequence and simultaneously contextualized by an individual cell's chromatin state.

Key Contributions

1. Unified multi-scale regulatory modeling. A shared sequence-based representation links DNA, chromatin accessibility, and cell-state-dependent gene expression through peak–peak self-attention and bidirectional, distance-biased peak–gene cross-attention.
2. Generalization beyond the training vocabulary. Held-out-region experiments across four paired datasets demonstrate expression prediction for genes and peaks unseen during task training, alongside transfer to unseen cell types.
3. Reusable regulatory representations. Enhancer–gene linking, motif-guided regulatory prioritization, zero-shot cell clustering, and few-shot adaptation assess transfer beyond expression prediction, with additional baselines and cross-task ablations to separate these gains from those of sequence embeddings alone.
4. Encoder-agnostic design. Replacing the frozen AlphaGenome features with Borzoi or NT-v3 embeddings tests whether the approach depends on any single pretrained sequence encoder.

Main Findings

• Gene expression prediction (seen sequences, seen cell types): MUSS reaches pseudobulk PCC of 0.909 on PBMC10K, 0.983 on BMMC, 0.996 on Openproblem, and 0.958 on Bone, the best or tied-best in all seen-sequence settings. The strongest non-sequence baseline, scMoGNN, reaches 0.851, 0.891, 0.996, and 0.955 respectively.
• Generalization to unseen sequences: With one quarter of genomic regions held out (including 1,161 of the initial 5,257 genes), MUSS achieves unseen-sequence, seen-cell-type PCC of 0.684 (PBMC10K), 0.748 (BMMC), 0.755 (Openproblem), and 0.683 (Bone). It exceeds scooby in PCC throughout, though scooby has higher unseen-sequence CCC on Bone (0.648 vs 0.611).
• Unseen cell types: MUSS obtains PCC of 0.698, 0.742, 0.762, and 0.683 across the four datasets in the unseen-sequence, unseen-cell-type setting.
• Best-or-tied coverage: MUSS attains the best or tied-best pseudobulk PCC in all seen-sequence settings and six of eight unseen-sequence settings.
• Single-cell resolution: Unseen-sequence PCCs at single-cell level are 0.436, 0.382, 0.535, and 0.410 across the four datasets; performance is competitive rather than uniformly best on seen sequences.
• Simple baselines remain strong in-distribution: PCA+Ridge reaches PCC 0.900 on PBMC10K and 0.972 on BMMC for seen genes, which the authors use to argue for testing beyond the training gene vocabulary.
• Enhancer–gene linking: Under a shared peak vocabulary and cosine-similarity protocol, MUSS outperforms AlphaGenome, Borzoi, GET, and EpiBERT across all distance bins and both GeneHancer evidence tiers (eQTL and CHi-C). In per-gene retrieval over a ±1 Mb cis window at roughly 1:36 class imbalance, MUSS reaches micro-AUPRC 0.1271 versus 0.0834 for Borzoi and 0.0532 for PCA+Ridge.
• Zero-shot cell clustering: On the held-out PBMC169K batch BIO_ddseq_1, MUSS leads all four metrics with ARI 0.5377 ± 0.0008, FMI 0.6973 ± 0.0006, NMI 0.6085 ± 0.0003, and AMI 0.6079 ± 0.0003, compared with ARI 0.4835 for ChromFound and 0.1032 for AlphaGenome.
• Ablations (BMMC, unseen sequence PCC): Removing sequence embeddings drops PCC to 0.196 (−0.549), and removing cross-scale interactions drops it to 0.009 (−0.736) — the largest architectural effect. Removing RBF distance bias gives 0.707 (−0.038), removing peak RoPE 0.725 (−0.020), removing distance reweighting 0.736 (−0.009), and removing peak MLM training 0.734 (−0.011).
• Sequence encoder choice: Across PBMC10K, BMMC, and Openproblem, AlphaGenome and Borzoi consistently outperform NT-v3 on unseen-sequence prediction, while differences on seen sequences are generally smaller. Borzoi is slightly higher on unseen PCC and R² for PBMC10K; AlphaGenome is best on all unseen-sequence metrics for BMMC and Openproblem.
• Few-shot adaptation: Starting from the same peak-MLM checkpoint on PBMC10K, PCC and CCC are approximately 0.5 with a single paired cell, improve with more cells, and begin to saturate after a few hundred. Both seen and unseen genes benefit.
• Motif-guided prioritization: Against CollecTRI (43,175 TF–gene interactions across 1,186 TFs) with FIMO motif detection, MUSS exceeds the distance baseline across the broad setting, with the largest gains at 250 kb–2 Mb. Under the conservative definition, pairwise AUROC at 50–100 kb is 0.641 ± 0.103 versus 0.519 ± 0.109 for distance alone.

Methodology in Plain English

Sequence-grounded entity encoding. Every regulatory element the model handles — each accessible peak and each gene — is defined by its coordinates on the human reference genome. MUSS extracts the DNA there and passes it through a frozen, pretrained AlphaGenome encoder, then averages the 128-bp-resolution embeddings across the interval to get one vector per peak or gene. Because both scales use the same frozen encoder, peaks and genes start in a shared DNA-grounded space. Input dimension is 3072, projected to a model dimension of 2048.

Two kinds of attention. Peaks attend to each other through full self-attention with rotary positional encodings applied using peak order, capturing dependencies among accessible regions. Genes and peaks then exchange information through stacked bidirectional cross-attention blocks (two blocks, 16 heads), so gene representations gather regulatory signal from peaks and peak representations are refined by gene context.

Distance as an explicit signal. Cross-scale attention carries an additive genomic-distance bias built from two parts. A Gaussian radial basis function encodes log(1+|d|) with learnable kernel centers and bandwidths, and a monotonic penalty −λ·log(1+|d|), with λ kept non-negative via softplus, reflects the decay of regulatory effects with distance. Inter-chromosomal pairs receive a constant bias, and a distance mask value of −35 is used.

Two-stage training. Stage one is peak masked language modeling: 15% of peaks per cell have their embeddings replaced with a learnable [MASK] token, and a head tied to the peak embedding matrix must recover the masked peak's identity from the contextualized representation. Stage two fine-tunes on paired scATAC-seq and scRNA-seq, predicting each gene's expression mean with a Poisson loss against observed counts.

Why unseen regions still work. New peaks and genes are encoded from DNA by the frozen encoder and passed through the same interaction layers; the MLM classification head is discarded at inference, so downstream prediction is not restricted to the training peak vocabulary.

Training setup. Pretraining uses PBMC169K (excluding the BIO_ddseq_1 and VIB_10xv1_1 batches) with Adam, learning rate 10⁻³, 100 warmup updates, effective batch size 32, five epochs, roughly 45 hours on four H200 GPUs. Fine-tuning uses learning rate 10⁻⁴ and dataset-specific epochs and batch sizes (30/8 for PBMC10K, 30/8 for BMMC, 15/8 for Openproblem, 5/16 for HCA Bone). Evaluation is on per-cell-type pseudobulk profiles using log₁₀(CPM+1) normalization, reported with PCC, CCC, and R².

Why This Matters

Impact on research. The paper argues that fixed-vocabulary single-cell models treat two peaks as independent tokens even when they share motif composition or cis-regulatory logic, and that sequence-to-function models such as Borzoi (524-kb inputs) and AlphaGenome (about 1 Mb) operate at contexts far shorter than global regulatory neighborhoods. MUSS offers a route to cell-resolved predictions that remain sequence-grounded, and its ablations show that both sequence information and learned accessibility context are needed — removing cross-scale interactions collapses unseen-sequence PCC to 0.009.

Real-world applications:

• Prioritizing candidate enhancer–gene links and TF–peak–gene relationships for follow-up CRISPR perturbation or enhancer engineering experiments.
• Zero-shot cell-type characterization from chromatin accessibility alone, without needing labels for the cell populations of interest.
• Data-efficient adaptation to new single-cell datasets: fine-tuning saturates after a few hundred paired cells, which suits settings where paired multiome data is scarce.
• Cross-omics imputation of expression from accessibility, including for genes and peaks not present during task training.

Industry relevance. The work was funded by Open Targets under OTAR 3090 through Drug2Cell, a project developing foundation models for scRNA-seq data, which signals direct interest from therapeutic target discovery. The authors caution that association scores are not causal or clinical evidence, and that unvalidated use in therapeutic targeting or sequence engineering could produce misleading conclusions or unintended biological effects.

Future Directions

• Scaling beyond quadratic attention. Exact peak self-attention remains quadratic in peak count even with memory-efficient kernels, and inference is slower than lightweight baselines; sparse or low-rank attention alternatives are proposed as future work.
• Reducing dependence on large frozen encoders. MUSS relies on a substantial pretrained sequence encoder and on paired multi-omic data for expression training, and alternative encoders differ in unseen-sequence accuracy.
• Extending enhancer–gene evaluation beyond the known vocabulary. The current linking evaluation is restricted to the training peak and gene vocabularies and does not establish generalization to unseen links.
• Disentangling the two directions of cross-scale flow. The upstream-only ablation does not isolate the two directions of peak–gene information flow, leaving the specific role of each direction an open question.

Target Audience

Researchers and practitioners in computational genomics and single-cell biology who work on regulatory modeling, multiome data integration, or sequence-based foundation models. It is most useful to readers already comfortable with transformer architectures and scATAC/scRNA analysis pipelines, and to groups evaluating whether sequence-grounded representations add value over fixed-vocabulary single-cell models or over simple baselines such as PCA+Ridge. The paper also matters to teams in drug target discovery and functional genomics who need candidate regulatory links to prioritize experimentally.

Authors’ abstract

Modeling gene regulation in single cells requires jointly representing DNA sequence, chromatin accessibility, and gene expression across regulatory scales. Sequence-based models capture DNA-level regulatory grammar but often lack cell-level resolution, whereas many single-cell omics models represent genes and peaks as fixed-vocabulary tokens without their underlying sequence. Here, we present MUSS, a sequence-based multi-scale model for single-cell gene regulation that preserves both DNA-level and cell-level resolution. MUSS derives peak- and gene-level representations from DNA using a frozen pre-trained sequence encoder and integrates them through peak-peak self-attention and distance-biased peak-gene interactions. Peak masked language modeling followed by training on paired scATAC-seq and scRNA-seq aligns these representations with cell-specific regulatory states. Across gene expression prediction, enhancer-gene linking, motif-guided regulatory prioritization, and zero-shot cell clustering, MUSS achieves strong performance against sequence-based and single-cell omics baselines. It further predicts expression for genes and peaks unseen during task training. These results support sequence-grounded, multi-scale representations for decoding cell-type-specific regulatory programs. Code, datasets, and model checkpoints are available at https://github.com/Lotfollahi-lab/MUSS_neurips26.