Evolutionary Curriculum Learning Improves Biological Sequence Modeling

AI summary

Overview

Research area: Machine learning for biological sequences — specifically training strategies for variational autoencoders (VAEs) built on multiple sequence alignments (MSAs), applied to protein variant effect prediction and RNA family sequence generation.

Technical level: Intermediate. The paper assumes familiarity with VAEs, MSAs, and basic statistics (paired t-tests), but its central idea — ordering training examples by evolutionary distance — is intuitive and stated in plain terms.

Scope: This paper introduces Evolutionary Curriculum Learning (ECL), a plug-in batch-sampling schedule for MSA-based biological sequence VAEs, and evaluates it on two architectures (EVE for proteins, RfamGen for RNA) across three proteins and three RNA families with five random seeds per configuration.

What This Paper Is About

Standard training of MSA-based biological VAEs treats every sequence in the alignment as interchangeable, drawing minibatches uniformly at random even though those sequences span a continuous range of evolutionary distances — from near-identical homologs to highly diverged ones. The authors argue this discards a natural hierarchy of learning difficulty, since evolutionarily close sequences offer locally consistent structural and functional constraints while distant sequences introduce much greater variation. ECL exploits that hierarchy by training on expanding neighborhoods of evolutionarily similar sequences around randomly chosen anchors, following a power-law schedule, with the goal of improving downstream tasks without changing model architecture.

Key Contributions

1. A curriculum-learning sampler for MSA-based VAEs. ECL replaces uniform minibatch sampling with an anchor-and-neighbors procedure: an anchor is drawn with probability proportional to phylogenetic sequence weights, and the remaining batch members are sampled from the anchor's k(t)-nearest neighbors, again weighted by phylogenetic weights.

2. A power-law neighborhood expansion schedule. The neighborhood size grows log-linearly from an initial k₀ to N−1 according to k(t) = ⌈k₀ · ((N−1)/k₀)^s(t)⌉ with s(t) = ((t−1)/(T−1))^γ, so early training stays inside a tight local neighborhood and expansion accelerates in the latter half of training; k(t) is clipped to [B−1, N−1].

3. Modality-specific evolutionary distance measures. Proteins use maximum-likelihood pairwise distances under WAG with a four-category discrete-Gamma rate model (α = 1.0); RNA uses a sum of normalized squared ℓ₂ distances across the three covariance-model feature components (transition rules, single-strand emissions, base-pair emissions), which the authors note recovers a per-position Hamming rate and lies in [0, 3].

4. Ablations isolating the source of the gain. A fixed-k variant (local sampling with no expansion) and a decoupled KL-annealing comparison show that the expansion, not locality alone and not the annealing schedule, drives the improvements.

Main Findings

• Protein variant classification improves on p53. Mean ClinVar AUROC for EVE rises from 0.9814 (baseline) to 0.9889 with ECL — a paired difference of +0.0075 (95% CI [−0.0012, +0.0161], t-test p = 0.075) — with ECL winning on 4 of 5 seeds.

• PTEN gains stability rather than central tendency. ECL reaches 1.0000 AUROC on every seed, while the baseline averages 0.9054 with four seeds at ≥ 0.99 and one collapsing to 0.5405. The paired difference (+0.0946) is not significant (p = 0.36) precisely because baseline variance is large. The authors frame this as a stability gain.

• The evaluation set is small and imbalanced. p53 retains 130 pathogenic and 27 benign variants; PTEN retains 111 pathogenic and only 2 benign. A 1.000 AUROC on PTEN means only that both benign variants rank below all 111 pathogenic ones. SCN5A leaves 23 labeled variants, all pathogenic, so AUROC is undefined and SCN5A appears only via training curves.

• A position-only baseline does not explain the results. A cross-validated k-NN classifier on residue index reaches AUROC 0.8414 on p53 and 0.3811 on PTEN, far below the VAE scores.

• Removing the expansion is actively harmful on p53. Fixed-k sampling drops mean AUROC to 0.9483, i.e. −0.0332 below baseline (95% CI [−0.0382, −0.0281], p = 0.0001), losing on all five seeds — evidence that the benefit depends on expansion rather than locality alone. On PTEN fixed-k also reaches 1.0000, but the authors call this uninformative because the metric is saturated.

• RNA generation improves in a consistent direction. Averaged over seeds, ECL raises mean CM bit scores on all three families: 86.0 vs. 83.0 for RF00169 (+3.7%, 5/5 seeds, p = 0.077); 150.2 vs. 139.6 for RF00234 (+7.5%, 3/5, p = 0.33); 153.7 vs. 152.3 for RF00638 (+0.9%, 4/5, p = 0.77).

• The pooled RNA statistic is significant only at the run level. ECL wins 12 of 15 seed-matched runs, which an exact binomial sign test rejects at p = 0.035. The authors explicitly caveat that the unit of inference is the training run, not the RNA family, and that a family-level test over three families cannot reach significance (a two-sided sign test over three families floors at p = 0.25).

• The curriculum sampler, not the annealing schedule, drives the result. Changing only the KL annealing (baseline + non-cyclic) lowers pooled mean bit score by 5.0% and hurts RF00234 (−11.5%) and RF00638 (−10.6%) while helping only RF00169 (+3.4%). ECL paired with the original cyclic schedule still improves (+5.7% pooled, 8/15 winning runs) versus +5.0% and 12/15 for the non-cyclic pairing.

• ECL is not delicately tuned, with one informative failure. On p53 every swept setting stays at or above baseline (k₀ ∈ {255, 512, 1024} → 0.9889, 0.9846, 0.9892; γ ∈ {1, 2, 4} → 0.9893, 0.9889, 0.9830), and pooled RNA improvement varies only between +4.7% and +6.1%. On PTEN, k₀ = 1024 drops mean AUROC to 0.7541 and wins on only 1 of 5 seeds, because PTEN has N−1 = 1076 sequences, so the initial neighborhood spans nearly the whole alignment and the curriculum effectively disappears.

• No per-step training overhead. Sampling an anchor and neighbors is O(B) per step, matching weighted uniform sampling. The only added cost is a one-time O(N²) pairwise distance matrix, computed once and reused across every run, seed, and hyperparameter setting.

• Training curves also favor ECL. ECL converges to lower validation negative ELBO and reconstruction loss than the baseline over 50,000 training steps on all three proteins, with particularly large gaps for p53 and PTEN.

Methodology in Plain English

The researchers first compute, once before training, the pairwise evolutionary distance between every pair of sequences in each alignment — using a WAG + Γ substitution model for proteins and covariance-model feature distances for RNA — and cache a sorted list of nearest neighbors for every sequence. Then, at each training step, instead of drawing a batch uniformly at random, they pick one anchor sequence and fill the rest of the batch from the anchor's nearest neighbors within a neighborhood whose size grows over training. The growth is multiplicative rather than additive: it starts at k₀ and reaches the entire alignment by the final step, so early steps are almost entirely local and the widening accelerates later. Sequence weights already used in the baselines (phylogenetic reweighting) are retained and reused for both anchor selection and neighbor sampling, so ECL sits on top of the existing reweighting rather than replacing it.

For evaluation, every configuration is trained with five random seeds (0–4), and each ECL variant is compared against the seed-matched baseline so all comparisons are paired. The authors report means with 95% confidence intervals, mean paired differences, paired t-test p-values, and the number of seeds won, while cautioning that n = 5 makes significance hard to reach and that the small, imbalanced evaluation sets make single-seed differences noisy. The one exception to the t-test is the pooled 15-run RNA aggregate, which uses an exact binomial sign test.

Why This Matters

Impact on research. The paper offers a low-cost, architecture-agnostic inductive bias: because ECL only reorders which sequences are visited and when, it composes with any MSA-based generative model and requires no new annotations, learned difficulty scores, or architectural changes. It also reframes MSA batch construction — normally treated as a neutral detail — as a design choice with measurable downstream effects, and its ablations (fixed-k, decoupled annealing) model a careful way of attributing gains when several training changes are bundled together.

Real-world applications:

• Clinical variant interpretation: distinguishing pathogenic from benign missense variants, as evaluated against ClinVar labels for p53 and PTEN.
• RNA engineering for therapeutics: generating novel functional RNA family sequences scored by the Rfam ground-truth covariance model.
• Functional RNA design pipelines that rely on sampling from a learned latent space structured around sequence and structure.
• Disease variant prediction more broadly, since the EVE-style approach underlies protein-level pathogenicity scoring.

Industry relevance. Pharma and biotech groups that run alignment-based generative models on proteins or RNA could adopt ECL without retraining pipelines from scratch, and the O(N²) preprocessing can be cached and reused across runs and seeds, which matters when many configurations are swept. The measured gains are modest and the authors do not claim state-of-the-art variant effect prediction, so the practical value is as an orthogonal add-on rather than a replacement for existing models.

Future Directions

1. Scale to larger benchmarks. The evaluation covers only three proteins and three RNA families, which the authors state is too few to support strong claims of generality; family-level RNA significance would require substantially more RNA families and compute.

2. Find cheaper distance proxies. The O(N²) preprocessing is the main obstacle to applying ECL to much deeper alignments. Because the curriculum only needs ordered neighbor lists rather than exact distances, any monotone proxy would suffice — identifying cheaper proxies is left to future work.

3. Test alternative evolutionary distance metrics. The authors suggest exploring distance measures beyond WAG + Γ for proteins and CM-feature distance for RNA.

4. Apply the local-to-global principle beyond VAEs. An open question is whether the same curriculum helps non-VAE sequence models.

Target Audience

Researchers and practitioners in computational biology and machine learning for biological sequences — particularly those training or deploying MSA-based generative models such as EVE and RfamGen — will benefit most. It is also useful for methodologists interested in curriculum learning, inductive biases from evolutionary data, and rigorous ablation design under small sample sizes. Readers need working familiarity with VAEs and sequence alignments; the core idea is accessible without deep phylogenetics background, though the distance definitions assume some comfort with substitution models and covariance models.

The paper's code is publicly available at https://github.com/KentoNishi/icml26-ecl.

Authors’ abstract

Variational autoencoders (VAEs) trained on multiple sequence alignments (MSAs) have emerged as powerful generative models for biological sequences, with applications ranging from disease variant prediction to functional RNA design. However, standard biological VAE training treats all sequences as exchangeable, ignoring the rich evolutionary structure that organizes homologous sequences from evolutionarily close to highly divergent. We propose Evolutionary Curriculum Learning (ECL), a training strategy that exploits this structure by progressively exposing the model to sequences of increasing evolutionary distance from sampled anchors, following a power-law expansion schedule. Applied to two architecturally distinct VAE models and two biological domains--protein variant effect prediction with EVE and RNA family sequence generation with RfamGen--ECL improves downstream task performance across five random seeds per configuration. Mean ClinVar classification AUROC rises from 0.981 to 0.989 for p53; for PTEN, ECL attains 1.000 in every seed whereas the baseline is unstable (mean 0.905, falling as low as 0.54). For RNA, ECL raises mean covariance-model bit scores on all three families tested and exceeds its seed-matched baseline in 12 of 15 training runs, though with only three families the effect cannot be established as significant at the family level. Ablation experiments show that progressively expanding the sampled sequences by evolutionary distance outperforms fixed-size neighborhood sampling in addition to uniform random sampling. Evolutionary distance is therefore a useful inductive bias for ordering the training curriculum in biological sequence modeling.