Static Recovery Is Not Dynamic Stability: Dynamics-Aware Benchmarking of Protein Motif Scaffolding

AI summary

Overview

Research area: Machine learning for protein design — specifically, benchmarking generative models that build protein scaffolds around functional motifs, with a new emphasis on molecular dynamics (MD) as an evaluation criterion.

Technical level: Advanced. The paper assumes familiarity with protein structure prediction, inverse folding, diffusion and flow-matching generative models, and all-atom molecular dynamics with free-energy landscape analysis (PCA/TICA).

Scope (one sentence): The authors build a larger, systematically derived motif-scaffolding benchmark (BILBY-PRO, 220 cases) and a dynamics-aware evaluation framework (BILBY-MD), then show that designs passing standard static refolding tests often fail to remain stable in explicit-solvent simulations.

What This Paper Is About

Protein motif scaffolding asks a generative model to construct a foldable protein around a small set of residues that carry biological function, such as an enzyme active site. Current benchmarks judge success by refolding a designed sequence with a structure predictor and checking whether the motif reappears in that single predicted structure — a static test. The authors ask whether that static test actually implies the design will hold its shape under realistic thermal motion, and whether the small hand-curated test sets used so far give a reliable picture of model performance.

Key Contributions

1. BILBY-PRO, a systematically derived benchmark of structurally conserved functional motifs built from PROSITE-linked experimental structures: 220 cases drawn from 174 unique PROSITE motif-pattern entries, with a median motif length of 19 residues. It is constructed independently of MotifBench but uses a compatible task format and evaluation protocol.

2. A multi-model static evaluation of six recent scaffold generators — RFDiffusion, Genie2, FrameFlow, Proteina, DPLM2 and ESM3 — on both the original 30-case MotifBench panel and BILBY-PRO, reported along two separate axes: target coverage and unique-solution yield (the MotifBench score).

3. BILBY-MD, a reference-based all-atom explicit-solvent MD evaluation framework that compares designs against simulations started from experimentally determined structures, measuring motif drift, scaffold drift, motif-contact preservation, and motif-dihedral free-energy surface agreement in PCA and TICA projections.

4. An empirical demonstration of misalignment between static benchmark success and dynamical fidelity: of 55 statically solved model–problem pairs, 31 (56.4%) meet the three-of-four MD-supported criterion and only 8 (14.5%) pass all four checks.

Main Findings

• Performance splits into coverage versus yield. RFDiffusion and Proteina solve more targets, whereas ESM3 produces more unique successful scaffolds on the targets it does solve. FrameFlow and Genie2 form an intermediate tier; DPLM2 performs weakly under both metrics.

• Models largely agree on which motifs are hard — except ESM3. Successful-solution counts correlate at r ≈ 0.61–0.96 among RFDiffusion, Proteina, FrameFlow, Genie2 and DPLM2 on the original MotifBench set, with r = 0.96 between RFDiffusion and Proteina, r = 0.85 between FrameFlow and RFDiffusion, and r = 0.87 between FrameFlow and Genie2. ESM3 correlates weakly with all others (r ≈ 0.06–0.25), and on BILBY-PRO the other models correlate at r ≈ 0.53–0.79 while ESM3 remains at r ≈ 0.11–0.26.

• Motif recovery survives MD better than scaffold stability does. Across solved pairs, 41/55 (74.5%) pass the motif-drift check but only 35/55 (63.6%) pass the scaffold-drift check. Mean motif RMSD differences fall within the 1 Å threshold for all six methods.

• Scaffold drift varies sharply by model. Mean scaffold RMSD differences were 7.46 Å (FrameFlow), 4.01 Å (Proteina), 3.56 Å (Genie2) and 2.54 Å (RFDiffusion) — all above the 2 Å threshold — while DPLM2 (−0.54 Å) and ESM3 (−0.40 Å) were slightly more stable than the reference.

• Contact preservation gives the strongest support for static solves. Only 10.5% of unsolved rows exceed the median solved row in contact Pearson similarity, and only 31.6% improve on the median solved row in weighted contact-map absolute difference. FrameFlow leads with Pearson 0.92 and weighted absolute difference 0.46; DPLM2, ESM3 and Genie2 average 0.84, 0.83 and 0.82; Proteina reaches 0.76; RFDiffusion, despite the broadest solved set, has Pearson 0.74 and weighted absolute difference 1.07.

• Free-energy landscapes expose failures that contacts miss. 27/55 (49.1%) of solved pairs pass the landscape-similarity check. Solved cases beat unsolved cases by a mean difference of 0.317 in PCA FES Pearson correlation and 0.279 in TICA FES Pearson correlation. Genie2, Proteina and DPLM2 are closer to the threshold in PCA but fall below it in TICA.

• Success is motif-dependent, not uniform. Compact, self-contained motifs hold up best: 06_6E6R (a 75-residue ferredoxin [2Fe–2S] motif, solved by all six models) has an MD support rate of 1.0, scaffold drift of 0.93 Å, and contact Pearson 0.91. 11_3TQB (dihydrofolate reductase) has support 0.75 with 0.40 Å scaffold drift and contact Pearson 0.96; 15_7A8S (a de novo TIM-barrel-derived design) has support 0.75 with 0.61 Å drift and contact Pearson 0.93.

• Local fidelity can coexist with large-scale relaxation. 09_7CG5 has support rate 1.0 with contact Pearson 0.97, PCA FES 0.85 and TICA FES 0.76, yet scaffold drift of 3.08 Å; 17_7DGW has contact Pearson 0.98 but scaffold drift of 7.89 Å.

• Interface and context-dependent motifs fail most often. 05_5ZE9 (V-ATPase P-loop/nucleotide-binding motif) is solved by five models but has an MD support rate of only 0.20; 04_5WN9 and 14_5IUS both have zero MD-supported solved pairs.

• Unsolved designs are clearly worse under MD, confirming the static label carries signal. In a control analysis of 79 statically unsuccessful pairs spanning 28 MotifBench problems, only 6/79 satisfy the three-of-four criterion (versus 31/55 for solved pairs) and none satisfy all four.

• Benchmark difficulty is class-dependent. Nucleic-acid-binding/transcription and membrane/transport cases are comparatively tractable, while metal/co-factor-binding and secreted defense/toxin/inhibitor cases are harder — though the authors caution that category-level conclusions are limited by small sample sizes.

Methodology in Plain English

The authors take two separate approaches.

Building a bigger, less arbitrary test set. Rather than hand-picking motifs, they start from PROSITE, a curated database of protein sequence patterns tied to families and functional sites. They match each pattern against experimentally solved structures in the Protein Data Bank, keep only motifs that appear in at least five structural instances, and require the median pairwise backbone RMSD across those instances to fall below a conservation threshold. They also flag "multi-state" motifs that recur in more than one structure using Foldseek-derived clustering. The result is 220 cases usable by existing scaffold-generation pipelines, each shipped with motif-only structures, source structures, segment annotations, redesignable positions and contig specifications.

Testing models the standard way, then testing them physically. For static evaluation they run six models under the conventional ProteinMPNN-plus-ESMFold refolding protocol, making 100 generation attempts per case and measuring both whether a case is ever solved (coverage) and how many unique successes it yields (MotifBench score). For the dynamics work they pick, for each solved MotifBench problem, the design with the highest TM score — a deliberately conservative choice — and simulate it for 200 ns in explicit solvent at 300 K, 1 bar and 150 mM NaCl, against 1 µs simulations of the corresponding experimental reference structures. They then ask four questions: did the motif stay put (motif RMSD drift within 1 Å), did the whole scaffold stay put (scaffold RMSD drift within 2 Å), did the motif's internal contacts persist, and did the motif's backbone dihedral angles explore the same free-energy landscape as the reference in both PCA and TICA projections. A pair counts as "MD-supported" if at least three of the four checks pass.

Why This Matters

The paper argues that a widely used benchmark metric is being read as stronger evidence than it is. A design that passes refolding-based motif recovery has been shown to be recoverable by one learned structure predictor from one sequence — not to be a physically stable object. Since the static pipeline is cheap and the MD check is expensive, the practical message is a screening hierarchy: use static metrics to filter, then confirm promising candidates with dynamics before treating them as viable.

Impact on research: The work separates three quantities that are usually collapsed into a single leaderboard — target coverage, unique-solution yield, and dynamical fidelity — and shows that model rankings can shift depending on which one is reported. It also gives the community a reproducible construction recipe for scaling motif benchmarks beyond hand-curated panels.

Real-world applications:

• Enzyme and biocatalyst engineering, where active-site geometry must persist through catalytic fluctuations rather than in one snapshot.
• Design of cofactor- and metal-binding sites, which the paper finds are among the more difficult and more variable classes.
• Binder and epitope-focused design, where the paper's results serve as a warning: interface-like motifs that are slices of larger binding surfaces performed worst under MD.
• Computational screening pipelines in protein therapeutics, where deciding which of many generated candidates deserves wet-lab follow-up is the central cost driver.

Industry relevance: Groups using RFDiffusion, ProteinMPNN, ESM3 or similar tools in production design loops can read this as a calibration study — a static "solved" label is reasonably trustworthy for local motif geometry but a weaker predictor of whole-scaffold stability, and models differ substantially in how much scaffold drift their solved designs show.

Future Directions

• Extending BILBY-MD to BILBY-PRO. The dynamics analysis in this paper covers only designs from the original MotifBench panel; the 220-case benchmark has not yet been subjected to MD evaluation.

• Longer and better-matched simulations. The 200 ns design trajectories versus 1 µs reference trajectories leave the comparison asymmetric, and the authors note that slower instabilities may be missed. Closing that gap, and moving beyond heuristic binary thresholds toward field-standard criteria, is an open methodological question.

• Incorporating discontinuous and multi-segment motifs. PROSITE patterns are sequence-level signatures biased toward contiguous or near-contiguous motifs, so multi-segment functional sites are under-represented. The authors identify multi-state motifs as informative cases for future dynamics-aware evaluation.

• Reducing dependence on the refolding oracle. The static pipeline's verdicts depend on ProteinMPNN and ESMFold, and cited prior work suggests evolutionary information can bias refolding-based assessments toward native-like structures even for substantially altered sequences. Predictor-independent success criteria remain unresolved.

Target Audience

Researchers and engineers working on generative protein design who use or build motif-scaffolding benchmarks; computational biophysicists interested in MD-based validation of designed proteins; benchmark designers in structural machine learning who want a concrete case study in how a success metric shapes conclusions; and practitioners in protein engineering or biotech who need to know how far a static "solved" label can be trusted before committing to experimental work. Readers without background in protein structure prediction or MD will find the paper dense, though the central claim — that passing a static test is not the same as being physically stable — is stated plainly enough to travel.

Authors’ abstract

Protein motif scaffolding benchmarks assess whether generative models can build scaffolds around functionally important residues. Their conclusions depend on which motifs are tested and which criteria define success. Existing benchmarks typically use small hand-curated motif sets and static refolding criteria, but protein function often depends on conformational ensembles and transitions. Here, we address both benchmark scope and success criteria. We introduce BILBY-PRO, a systematically derived benchmark of structurally conserved functional motifs from PROSITE-linked experimental structures, comprising 220 cases from 174 PROSITE motif-pattern entries. Evaluating six recent scaffold-generation models reveals distinct patterns in target coverage and unique-solution yield: RFDiffusion and Proteina solve more targets, whereas ESM3 produces more unique successful scaffolds on the targets it solves. We further introduce BILBY-MD, a reference-based all-atom molecular dynamics evaluation framework. Applied to statically solved designs from the original MotifBench benchmark, it compares motif drift, scaffold drift, motif-contact preservation, and motif-dihedral free-energy landscapes against reference simulations. Of 55 statically solved model-problem pairs, 31 satisfy our MD-supported criterion and only 8 pass all checks. Compact, self-contained motifs are often MD-supported, whereas interface-like or context-dependent motifs fail more frequently. Static refolding success therefore provides useful evidence of local motif recovery but does not establish dynamical fidelity of the full scaffold. BILBY highlights three complementary axes of motif-scaffolding evaluation: target coverage, unique-solution yield, and MD-supported dynamical fidelity.