Augmenting Biological Fitness Prediction Benchmarks with Landscapes Features from GraphFLA

AI summary

Overview

Research area: Machine learning for biological sequence-fitness prediction, evolutionarily informed benchmark analysis, and fitness landscape topography.

Technical level: Intermediate. The paper is about a software framework and benchmark augmentation; readers need some familiarity with protein/RNA/DNA mutational datasets, but the landscape concepts are explained in plain terms.

Scope: The paper introduces GraphFLA, a scalable Python framework that builds fitness landscapes from mutagenesis data and computes 20 topographical features, then uses those features to explain and compare the performance of fitness prediction models across ProteinGym, RNAGym, and CIS-BP.

What This Paper Is About

Benchmarks like ProteinGym and RNAGym contain hundreds of tasks, but they describe each task only with basic labels (such as taxon) or simple statistics (such as sequence length). Because one model rarely dominates everywhere, users fall back on average scores, which hides why a model succeeds on one set of tasks and fails on another. GraphFLA fills this gap by computing quantitative descriptions of the underlying fitness landscape — ruggedness, epistasis, navigability, and neutrality — for each benchmark task, turning those descriptions into interpretable meta-features that explain model behaviour.

Key Contributions

1. A scalable landscape analysis framework. GraphFLA is a Python framework that preprocesses sequence-fitness data, constructs the landscape as a directed variant network, and computes 20 biologically grounded features spanning 4 topographical aspects (ruggedness, navigability, epistasis, neutrality). It is designed for applicability, interoperability with ML-ready data, scalability to millions of variants, and extendability.

2. A large validated dataset release. The authors release 155 combinatorially complete empirical fitness landscapes covering over 2.2 million variants (55 DNA datasets with 724k mutants, 63 protein datasets with 1.1M mutants, 37 RNA datasets with 348k mutants), collected from 61 works, plus 5 theoretical models for generating synthetic landscapes.

3. Large-scale application to existing benchmarks. GraphFLA constructs more than 5,300 landscapes from ProteinGym (217 DMS substitution tasks), RNAGym (31 landscapes), the 155 combinatorial landscapes, and CIS-BP (5,016 transcription factor binding landscapes over 174M total mutants).

4. Evidence that landscape features explain model performance. The paper shows that landscape topography predicts how well models perform and where different models hold advantages, answering both of the motivating questions.

Main Findings

• GraphFLA is dramatically faster and lighter than prior tools. On a single core of an Intel Xeon Platinum 8260 CPU with 256GB RAM, the two baselines took more than 5 hours and more than 3 hours respectively to construct a landscape at the scale of 1 million mutants, whereas GraphFLA took 20 seconds. GraphFLA required only 2GB of memory for 1 million mutants, while the community implementation hit out-of-memory errors beyond 100,000 mutants. The prior package MAGELLAN is written in C and handles landscapes around 10^5 variants; GraphFLA scales to 10^7.

• Analysis reliability was validated by replication. Across the 61 papers supplying the 155 combinatorial datasets, GraphFLA replicated the qualitative conclusions of all 61 studies. For features with unique definitions such as the fraction of local optima and reciprocal sign epistasis, GraphFLA reproduced published values when data processing details were sufficiently described.

• Feature calculations are robust to imperfect data. On a complete NK landscape with n=15, k=7, reciprocal sign epistasis stayed near the reference value of 0.1885 even with 50% of variants removed, with noise up to 0.2σ, and under biased sampling. Global optima accessibility was the exception, falling from 0.6729 in the reference to 0.3223 when 50% of data was removed, an effect the authors say is predictable and can be partially corrected by scaling by the fraction of remaining data (1−α).

• Landscape diversity is substantial. Across the 155 combinatorial landscapes, the fraction of local optima ranges from around 0% to 5%; the low end is a fairly smooth, unimodal landscape, and the high end is on par with the most rugged NK landscapes.

• Hard landscapes share recognizable traits. Using Evo2-7b (trained on 9.3 trillion DNA base pairs), the authors found performance varies strongly with landscape features: half of the 20 features showed Spearman's |ρ| above 0.6 with Evo2's performance, and 6 showed moderate correlation (0.3 < |ρ| < 0.6). Models struggle on landscapes that are more rugged and more epistatic, less navigable, and highly neutral.

• Performance is task-dependent even for strong models. VenusREM has the highest average score across all 217 ProteinGym DMS substitution tasks but leads on only 14 (6.5%) individual tasks, and 44 of the evaluated 89 models lead on at least one task.

• Feature space separates easy from hard tasks. On the 5,016 CIS-BP transcription factor binding landscapes, a quadrant with both a large number of local optima and abundant non-magnitude epistasis was hard for Evo2 to rank, while a quadrant with low values of both typically yielded Spearman's ρ > 0.5.

• Landscape features explain model-vs-model gaps. VenusREM and ProSST (k=2,048) have similar zero-shot ProteinGym scores (Spearman's ρ of 0.518 vs 0.507) and lead on 53% and 47% of tasks respectively. On benign landscapes with little reciprocal sign epistasis (ε_reci < 0.1), ProSST tends to outperform VenusREM (Wilcoxon signed-rank test, w=123, p=0.003), but for landscapes with ε_reci > 0.15, VenusREM consistently outperforms ProSST. Both zero-shot models only beat the supervised Kermut on highly navigable landscapes (FDC ≈ −0.7); on the ODP2 landscape with FDC = 0.23, Kermut outperforms VenusREM by a Spearman's ρ of 0.53.

• Directed evolution outcomes track landscape topography. On 20 protein landscapes that are 3- or 4-site saturated (20^3 or 20^4 variants), basic greedy adaptive walk DE finds near-optimal variants on benign landscapes but struggles on rugged, epistatic, less navigable ones. Five ML- or active-learning-guided approaches were also susceptible to epistasis, but MLDE with a zero-shot warm start and the ALDE variants were more robust. Each approach was aggregated over 100 randomly initialized runs, with ALDE iterations set to 3 or 5 rounds.

• Broader applicability. GraphFLA was demonstrated on 6 microbial community-function landscapes and on phenotype landscapes (genotype-phenotype maps) for RNA secondary structure, protein tertiary structure, and protein complexes, using 3 phenotype landscape features.

Methodology in Plain English

The framework has three stages. First, data preprocessing takes the standard machine learning inputs — a list of biological sequences and their fitness values — and standardizes them, detects the composition of the sequence space, and identifies duplicates or missing values. It supports arbitrary alphabets at each locus, so it is not limited to DNA, RNA, or protein.

Second, landscape construction identifies which variants are genetic neighbours. Rather than computing pairwise genetic distances between all variants, which needs quadratic time and memory, GraphFLA directly generates all potential single-mutation neighbours for each variant, achieving nearly linear complexity. The landscape is then stored as a directed graph where nodes are variants carrying fitness values and directed edges point toward the fitter variant of a neighbouring pair. Because this is backed by the igraph package in C, classical analyses become graph-mining problems — local optima are the sinks of the graph, and epistasis types correspond to specific 4-node motifs.

Third, landscape analysis computes 20 features across 4 aspects: ruggedness (such as fraction of local optima, autocorrelation, gamma statistic), epistasis (magnitude, sign, reciprocal sign, diminishing returns, and others), navigability (fitness-distance correlation, global optima accessibility, basin-fitness correlation, evolvability), and neutrality. These 20 features were selected via a large language model-assisted, data-driven survey of 1,673 papers, narrowing more than 100 initial candidates down by literature frequency, biological significance, coverage, computational feasibility, and compatibility.

To validate the tool, the authors benchmarked runtime and memory against MAGELLAN and a community implementation using NK-model landscapes with N from 5 to 20 and averages over 10 replicates, replicated published conclusions from 61 studies, tested robustness on synthetic landscapes with missing, noisy, and biasedly sampled data, and then applied the features to real benchmarks alongside model performance measured by Spearman's ρ. Correlations were also checked using partial correlations controlling for landscape size.

Why This Matters

Impact on research. Average benchmark scores can be misleading — as the paper shows, a model with the best average can lead on only 6.5% of individual tasks. Landscape features give researchers a biologically meaningful way to explain why a model wins or loses on a task, to compare models along dimensions that matter for a given application, and to choose models for landscapes with particular properties. Because the framework is also helpful for directed evolution, it connects prediction benchmarking to protein engineering practice.

Real-world applications:

• Protein engineering and drug development. Choosing whether a zero-shot or supervised model is appropriate for a target depends on whether the target's fitness landscape is navigable or epistatic, which GraphFLA can quantify in advance.
• Variant effect and disease prediction. Landscape features can indicate when a model's predictions on mutational effects are likely to be unreliable, since neutrality and epistasis make effects hard to extrapolate.
• Directed evolution campaigns. The released combinatorial landscapes and the analysis of 5 DE and ML-guided methods indicate which optimisation strategies are robust on rugged versus benign landscapes.
• Transcription factor and regulatory analysis. The 5,016 CIS-BP binding landscapes with 174M mutants provide a large resource for studying DNA binding specificity across 329 eukaryotic species.

Industry relevance. Cell and gene therapy, enzyme engineering, antibody optimisation, and agricultural biotechnology all rely on predicting mutational effects; a tool that reveals when predictions can be trusted and which model to use on a given landscape has direct practical value. The Python API and interoperability with ML-ready data also lower the barrier to adoption inside existing model-training pipelines.

Future Directions

• Extending beyond the current feature set. GraphFLA is designed to be extendable via a unified API, and the authors note there is no consensus on a comprehensive feature set, leaving room for new metrics.
• Broader biological scales. The paper sketches applications to gene regulatory networks, metabolic landscapes at the cellular level, and community-composition landscapes, but only demonstrates 6 microbial community-function landscapes.
• Better handling of data incompleteness. Global optima accessibility degraded with missing data; a principled correction beyond the suggested scaling by (1−α) remains open.
• Clarifying model choice guidance. The findings show when supervised models outperform zero-shot ones and vice versa, but turning these patterns into prescriptive recommendations for new tasks is a natural next step.

Target Audience

Researchers and practitioners working on machine learning for biological sequences, benchmark designers for ProteinGym- or RNAGym-style evaluations, protein and RNA engineers who use directed evolution or ML-guided directed evolution, computational biologists studying fitness landscapes and epistasis, and bioinformatics tool developers who need a scalable, Python-native alternative to existing landscape analysis software.

Authors’ abstract

Machine learning models increasingly map biological sequence-fitness landscapes to predict mutational effects. Effective evaluation of these models requires benchmarks curated from empirical data. Despite their impressive scales, existing benchmarks lack topographical information regarding the underlying fitness landscapes, which hampers interpretation and comparison of model performance beyond averaged scores. Here, we introduce GraphFLA, a Python framework that constructs and analyzes fitness landscapes from mutagensis data in diverse modalities (e.g., DNA, RNA, protein, and beyond) with up to millions of mutants. GraphFLA calculates 20 biologically relevant features that characterize 4 fundamental aspects of landscape topography. By applying GraphFLA to over 5,300 landscapes from ProteinGym, RNAGym, and CIS-BP, we demonstrate its utility in interpreting and comparing the performance of dozens of fitness prediction models, highlighting factors influencing model accuracy and respective advantages of different models. In addition, we release 155 combinatorially complete empirical fitness landscapes, encompassing over 2.2 million sequences across various modalities. All the codes and datasets are available at https://github.com/COLA-Laboratory/GraphFLA.