BEND 2.0: Benchmarking the Capabilities Needed to Design DNA

AI summary

Overview

Research area: Computational genomics and machine learning — specifically benchmarking genomic foundation models on whether they can support the design of DNA sequences, not just describe them.

Technical level: Intermediate. Readers need some familiarity with machine learning evaluation (linear probes, Spearman correlation, held-out splits) and basic molecular biology (regulatory elements, splicing, expression), but the paper is organized around conceptual capabilities rather than architecture details.

Scope: BEND 2.0 is a benchmark of eight tasks across five experimental datasets that measures five capabilities — sequence priors, functional prediction, biological context, genetic interactions, and genomic design — in genomic foundation models.

What This Paper Is About

Genomic foundation models are increasingly asked to propose DNA sequences and edits, but existing benchmarks mostly measure how well a model describes sequence or predicts aggregate functional readouts — not whether its predictions can actually guide a design choice. The authors build BEND 2.0, an extension of the earlier BEND benchmark, to test the specific predictive distinctions that matter for design: can a model separate a mutation's average effect from its cell-type-specific effect, can it predict how one mutation changes another's effect, and can its rankings pick a sequence that produces a desired response? The benchmark spans eight tasks built on five experimental datasets and evaluates a wide roster of models, from DNA language models (Evo2 20B, DNABERT-2, GENERATORv2, CENO 1B, NTv3 at 100M and 650M) to functional models (AlphaGenome, Enformer, SpliceAI) plus simple baselines.

Key Contributions

1. A five-capability evaluation framework. BEND 2.0 formalizes Prior, Function, Context, Interaction, and Design as distinct measurable capabilities, each with its own task construction and metric, and separates effects into mean versus context-specific deviation and additive versus interaction components.

2. A systematic separation of "native outputs" from "fitted linear readouts." The paper compares what a model outputs through its own heads against ridge regression heads fitted to frozen embeddings (with the representation change between variant and reference as the feature), showing these two access routes give meaningfully different rankings.

3. Interaction-specific scoring. The benchmark scores the non-additive residual of double mutations separately from the combined effect, and quantifies how much predicted interactions actually contribute to overall accuracy by ablating them.

4. Design tasks with experimentally measured outcomes. Rather than scoring predictions, the design tasks rank real candidate edits and sequences in assayed libraries (with candidate measurements withheld from selection) and report the measured utility of the model's choices, benchmarked against an experimental selection reference.

Main Findings

• Model choice and representation access both change results. Linear readouts fitted to frozen embeddings can match or exceed models' native outputs on several tasks. For example, AlphaGenome's fitted head reaches 0.650 versus 0.621 for its native outputs on OpenSplice splicing effects, and NTv3 650M post's fitted head reaches 0.564 versus 0.461 for native outputs.

• Functional post-training substantially improves capabilities. On OpenSplice, NTv3 650M goes from 0.187 pre-trained to 0.564 post-trained; on the Siraj cell-contrast task it goes from 0.010 to 0.256; on tissue identification from 7.8% to 56.9%.

• Sequence-prior scores depend on how the model is queried. Hiding only the central base of a 1,200 bp window gives NTv3 650M post a loss of 2.199 bits; masking an additional 12% reduces this to 1.375, and averaging over eight masking draws reduces it further to 1.346. A fixed shift in internal activations raised accuracy from 29% to 56% without changing weights, close to the pre-trained model's 55%.

• Simple calibration helps too. Temperature calibration lowers GENERATORv2's loss from 2.168 to 1.711 while leaving accuracy at 50.2%.

• Sequence priors do not imply functional understanding. Evo2 and CENO reconstruct 55.2% and 62.4% of assay reference bases, yet their allele scores correlate only 0.045 and 0.029 with reporter effects.

• A specialist model still leads on splicing. SpliceAI reaches 0.674 on OpenSplice mutation effects, above AlphaGenome's 0.650 and NTv3 650M post's 0.564. A readout from the matched architecture with random weights reaches 0.144, and a task-trained one-hot ResNet reaches 0.488.

• Readouts transfer across assays and genes. OpenSplice-trained readouts evaluated on 4,618 COMPASS mutations across 88 genes excluded from source training (fitted without COMPASS labels) retain substantial accuracy and outperform a k-mer baseline fitted on the OpenSplice training set.

• Average effects are predicted better than context-specific ones. On 128 Siraj regulatory elements, AlphaGenome predicts the mean mutation effect at 0.532 versus 0.398 for the HepG2–K562 difference; NTv3 650M post reaches 0.492 and 0.256. The gap is larger for NTv3.

• Tissue-level context is tractable but unsaturated. On 5,782 held-out genes across 51 GTEx tissues, AlphaGenome reaches 68.6% tissue identification (random chance 2.0%, k-mer ridge 7.8%, experimental reference 99.0%); NTv3 650M post reaches 45.1%.

• Combined-effect accuracy can conceal weak interaction prediction. On COMPASS's 110-gene panel, NTv3 650M post reaches 0.409 for single effects and 0.500 for double effects but only 0.046 for the interaction; replicate correlation for the interaction measurements themselves is 0.171. Removing the predicted interaction changes the double-mutation correlation only from 0.500 to 0.493.

• Background interactions are predicted weakly. Across 68,026 paired mutations in 114 regulatory elements, AlphaGenome reaches 0.260, NTv3 650M post 0.187, and Enformer 0.181; accuracy is highest for mutations within 10 bp of each other.

• Models help narrow candidate edits, but leave room for improvement. For target matching across 331 targets in 110 elements, AlphaGenome's first choice has an average percentile rank of 24.3 and NTv3 650M post's 30.0, versus 50 for uniform selection; at least one of the top ten choices matches within assay uncertainty for 62% (AlphaGenome) and 59% (NTv3) of targets versus 38% under uniform selection.

• Cell-selective selection works but trails experimental selection. At equal weighting, AlphaGenome reaches measured utility 0.384, NTv3 650M post 0.287, NTv3 100M post 0.211, and no edit 0. Linear readouts fitted to AlphaGenome's frozen embeddings reach 0.515. Experimental selection on the same evaluation subsets outperforms native AlphaGenome by 0.603 in utility, with higher utility at 108 of 127 loci.

• Multi-cell design remains hardest. Gains in utility over random selection are 0.707 (AlphaGenome), 0.700 (Enformer), 0.482 (NTv3 650M post), and 0.344 (NTv3 100M post), but mean utility stays negative at −0.222 for AlphaGenome and −0.228 for Enformer. A linear readout on AlphaGenome's frozen embeddings raises the gain to 0.833 with mean utility still −0.096, while selecting with one repeat measurement and evaluating on the other gives +0.673.

Methodology in Plain English

The authors assemble experimental datasets where the same DNA sequences or edits were measured in the lab, then hold out whole biological units — genes, regulatory elements, loci, or sequence families — so the model is always tested on material it was not fitted on. For each task they compare two ways of getting a prediction out of a model: the model's own output heads (native outputs), and a simple linear model fitted on top of the model's frozen internal representation of the sequence (a linear readout). For mutation effects, the readout is fitted on the difference between the variant's and the reference's representation, which isolates what the edit changed.

Each capability gets a targeted test. Sequence priors are scored as bits per base for reconstructing a masked base in 1,200 bp windows. Functional prediction uses exon-inclusion changes from OpenSplice and then transfers the learned readout to a different assay (COMPASS) without refitting. Biological context uses the Siraj dataset, where identical mutations are measured in two cell lines, and splits a measured effect into its mean across contexts plus a context-specific deviation; GTEx expression across 51 tissues adds a tissue-identification test. Interactions use the non-additive residual of double mutations, measured both across allelic backgrounds and within COMPASS.

The design tasks are the distinctive part. Models rank real candidate edits or enhancers drawn from libraries that were actually assayed, with the measurements hidden until scoring. Utility functions encode the goal — matching a reference edit's effect, increasing expression in a target cell while penalizing change in another (U = t − λ|o|), or rewarding the weakest ON cell while penalizing the strongest OFF cell. The resulting choices are then scored against the withheld measurements, and compared against an experimental reference that uses one set of repeat measurements to choose and another to evaluate. Uncertainty is estimated by bootstrapping whole biological groups, and correlations are computed within each group and averaged so no single large gene dominates.

Why This Matters

The paper shifts benchmark design from "does the model predict well on average" to "does the model make the distinctions a designer needs." It shows that aggregate accuracy can hide failures — a model can predict a double mutant's effect well while capturing almost none of the interaction, and can predict a mutation's mean effect well while being much weaker on the difference between cell types. It also shows that functional post-training and simple representation-level interventions change results enough that how a model is queried is itself a research variable.

Real-world applications:

• Regulatory sequence and enhancer design for cell-type-specific expression, including the multi-cell ON/OFF patterning task tested here against Atlas-designed enhancers.
• Variant prioritization in genetics, where distinguishing a mutation's average effect from its cell-type-specific or background-specific effect determines which candidates are worth assaying.
• Therapeutic and synthetic biology workflows such as engineering cell-type-targeted activity or, as cited in the paper, viable bacteriophage genomes — where reducing the number of candidates to test has direct cost and time value.
• Model development and selection for labs choosing between pretrained genomic models, since the benchmark reports native-output and fitted-readout performance separately.

Industry relevance: the negative-utility results on multi-cell design quantify a gap between what current models promise and what they deliver for multi-objective sequence design, which is directly relevant to groups building or procuring genomic design tools. The finding that linear readouts on frozen embeddings rival native heads is also practically useful for teams that want to adapt a model without retraining it.

Future Directions

• Extending strong sequence priors to natural haplotypes. The main prior evaluation uses GRCh38; the authors explicitly flag this as an important next step based on their appendix results.

• Improving context-specific and interaction prediction. These are the capabilities where aggregate accuracy most clearly overstates model understanding, and where the paper's ablation shows predicted interactions contribute very little to overall accuracy.

• Better multi-objective design methods. The multi-cell task has negative mean utility for every model tested, while the experimental selection reference reaches +0.673, so substantial headroom exists within the same sequence bank.

• Accounting for benchmark limitations. The authors note that experimental references are somewhat naive and can still beat models, that linear-head results may depend on embedding size, and that differing input context lengths may affect task performance — all of which bound how the results should be interpreted.

Target Audience

Machine learning researchers working on genomic foundation models and their evaluation; computational biologists and geneticists who need to choose a model to prioritize wet-lab candidates; benchmark designers interested in how to separate native outputs from fitted readouts and in how to score non-additive effects; and industry teams in synthetic biology or therapeutic design who need a realistic picture of where current genomic models help and where they fall short.

Authors’ abstract

Genomic foundation models are increasingly used to propose DNA sequences and edits. But successful design requires capabilities that current benchmarks do not measure. Here we introduce BEND 2.0 (BENchmark for DNA language models), which tests how well models predict the effects of DNA changes and whether these predictions can guide sequence design. Across eight tasks built on five experimental datasets, we find that model choice and how representations are accessed both affect performance. Linear readouts fitted to frozen representations can match or exceed models' native outputs, and functional post-training substantially improves the models' capabilities. Linear readouts transfer well across experimental assays and genes, suggesting that models capture functional patterns beyond the data used to fit individual predictors. At the same time, strong prediction of average effects can conceal weaker prediction of context-specific effects, and accurate prediction of combined mutation effects does not imply accurate prediction of their interactions. In design tasks, model predictions can reduce the number of candidates needed to find a desired response, but still leave substantial room for better design choices, particularly for context-specific and multi-cell objectives. Our benchmark reveals that genomic models now display signs of general utility but have yet to saturate on performance in even simple tasks.